2024-04-23T03:14:42.815 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-23T03:14:42.815 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --name teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} --archive /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734 --verbose -- /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/orig.config.yaml 2024-04-23T03:14:42.844 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T03:14:42.930 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734 branch: main description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7668734' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 workunit: branch: main sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 6458 sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 sleep_before_teardown: 0 subset: 39/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 8254cb5a247caa8e3d35e534a1fefe2362a0a4b8 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEjLpAwTZqGo+f69gG/J23ZYHahwEZJxuD/yaed9FhB297Avs9YMRhGQ+kf1nRdmGz4PSGxcunfzdv2PPBgcrb8= smithi149.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE/dledikhDHGIDwcvDkB4g9yGeG5dsOjTgrNpdLsrc4ksWDm22WN6lerGXkKkYEvqMPVA8XtBiuN6292PY8PA= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-23T03:14:42.930 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/qa; will attempt to use it 2024-04-23T03:14:42.930 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_8254cb5a247caa8e3d35e534a1fefe2362a0a4b8/qa/tasks 2024-04-23T03:14:42.930 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-23T03:14:42.932 INFO:teuthology.task.internal:Checking packages... 2024-04-23T03:14:42.954 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '56e81df3aeb98b717efe2ab0537a8c60249f95f7' 2024-04-23T03:14:42.954 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-23T03:14:42.954 INFO:teuthology.packaging:ref: None 2024-04-23T03:14:42.954 INFO:teuthology.packaging:tag: None 2024-04-23T03:14:42.954 INFO:teuthology.packaging:branch: main 2024-04-23T03:14:42.954 INFO:teuthology.packaging:sha1: 56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:14:42.954 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-23T03:14:43.178 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3212.g8254cb5a 2024-04-23T03:14:43.179 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-23T03:14:43.186 INFO:teuthology.task.internal:no buildpackages task found 2024-04-23T03:14:43.187 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-23T03:14:43.194 INFO:teuthology.task.internal:Saving configuration 2024-04-23T03:14:43.207 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-23T03:14:43.213 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T03:14:43.237 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 03:09:34.868345', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEjLpAwTZqGo+f69gG/J23ZYHahwEZJxuD/yaed9FhB297Avs9YMRhGQ+kf1nRdmGz4PSGxcunfzdv2PPBgcrb8='} 2024-04-23T03:14:43.256 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 03:09:34.870034', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE/dledikhDHGIDwcvDkB4g9yGeG5dsOjTgrNpdLsrc4ksWDm22WN6lerGXkKkYEvqMPVA8XtBiuN6292PY8PA='} 2024-04-23T03:14:43.256 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-23T03:14:43.262 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-23T03:14:43.262 INFO:teuthology.task.internal:roles: ubuntu@smithi149.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-23T03:14:43.262 INFO:teuthology.run_tasks:Running task console_log... 2024-04-23T03:14:43.326 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd8730470d0>, signals=[15]) 2024-04-23T03:14:43.327 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-23T03:14:43.333 INFO:teuthology.task.internal:Opening connections... 2024-04-23T03:14:43.334 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-04-23T03:14:43.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T03:14:43.417 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi149.front.sepia.ceph.com 2024-04-23T03:14:43.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T03:14:43.490 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-23T03:14:43.498 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-04-23T03:14:43.516 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-04-23T03:14:43.517 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T03:14:43.575 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-04-23T03:14:43.576 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T03:14:43.576 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T03:14:43.576 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T03:14:43.576 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-04-23T03:14:43.597 DEBUG:teuthology.orchestra.run.smithi149:> uname -m 2024-04-23T03:14:43.614 INFO:teuthology.orchestra.run.smithi149.stdout:x86_64 2024-04-23T03:14:43.615 DEBUG:teuthology.orchestra.run.smithi149:> cat /etc/os-release 2024-04-23T03:14:43.671 INFO:teuthology.orchestra.run.smithi149.stdout:NAME="CentOS Stream" 2024-04-23T03:14:43.671 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION="9" 2024-04-23T03:14:43.671 INFO:teuthology.orchestra.run.smithi149.stdout:ID="centos" 2024-04-23T03:14:43.671 INFO:teuthology.orchestra.run.smithi149.stdout:ID_LIKE="rhel fedora" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:VERSION_ID="9" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:PLATFORM_ID="platform:el9" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:ANSI_COLOR="0;31" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:LOGO="fedora-logo-icon" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:HOME_URL="https://centos.org/" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-23T03:14:43.672 INFO:teuthology.orchestra.run.smithi149.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-23T03:14:43.672 INFO:teuthology.lock.ops:Updating smithi149.front.sepia.ceph.com on lock server 2024-04-23T03:14:43.697 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-23T03:14:43.706 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-23T03:14:43.710 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-23T03:14:43.711 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-04-23T03:14:43.713 DEBUG:teuthology.orchestra.run.smithi149:> test '!' -e /home/ubuntu/cephtest 2024-04-23T03:14:43.728 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-23T03:14:43.734 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-23T03:14:43.734 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-04-23T03:14:43.770 DEBUG:teuthology.orchestra.run.smithi149:> test -z $(ls -A /var/lib/ceph) 2024-04-23T03:14:43.793 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-23T03:14:43.844 INFO:teuthology.run_tasks:Running task kernel... 2024-04-23T03:14:43.858 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-23T03:14:43.858 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-23T03:14:43.859 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T03:14:43.859 DEBUG:teuthology.orchestra.run.smithi149:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-23T03:14:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:14:43.874 DEBUG:teuthology.orchestra.run.smithi149:> uname -r 2024-04-23T03:14:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:14:43.876 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-04-23T03:14:43.930 INFO:teuthology.orchestra.run.smithi149.stdout:5.14.0-437.el9.x86_64 2024-04-23T03:14:43.930 INFO:teuthology.task.kernel:Running kernel on smithi149: 5.14.0-437.el9.x86_64 2024-04-23T03:14:43.930 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum install -y kernel 2024-04-23T03:14:43.934 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-437.el9.x86_64 2024-04-23T03:14:43.934 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-437.el9.x86_64 2024-04-23T03:14:43.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-04-23T03:14:48.051 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-04-23T03:14:50.290 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-04-23T03:14:50.401 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.0 MB 00:04 2024-04-23T03:14:58.309 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 1.5 MB/s | 6.2 MB 00:04 2024-04-23T03:15:00.346 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 16 kB 00:00 2024-04-23T03:15:02.589 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 9.9 MB/s | 21 MB 00:02 2024-04-23T03:15:07.814 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-23T03:15:09.745 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T03:15:09.746 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T03:15:09.842 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-23T03:15:09.843 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-04-23T03:15:09.843 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-04-23T03:15:09.946 DEBUG:teuthology.orchestra.run.smithi007:> echo no | sudo yum reinstall kernel || true 2024-04-23T03:15:10.472 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 03:15:07 AM UTC. 2024-04-23T03:15:10.595 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-23T03:15:10.606 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-23T03:15:10.606 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-04-23T03:15:10.606 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout:Reinstalling: 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 255 k 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 0 2024-04-23T03:15:10.607 INFO:teuthology.orchestra.run.smithi007.stderr:Operation aborted. 2024-04-23T03:15:10.662 INFO:teuthology.orchestra.run.smithi007.stdout:Is this ok [y/N]: 2024-04-23T03:15:10.663 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum reinstall -y kernel || true 2024-04-23T03:15:11.181 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:04 ago on Tue 23 Apr 2024 03:15:07 AM UTC. 2024-04-23T03:15:11.301 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout:Reinstalling: 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:15:11.302 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-04-23T03:15:11.303 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-23T03:15:11.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:15:11.303 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 255 k 2024-04-23T03:15:11.303 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 0 2024-04-23T03:15:11.303 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-04-23T03:15:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-437.el9.x86_64.rpm 855 kB/s | 255 kB 00:00 2024-04-23T03:15:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-04-23T03:15:11.783 INFO:teuthology.orchestra.run.smithi007.stdout:Total 530 kB/s | 255 kB 00:00 2024-04-23T03:15:11.804 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-04-23T03:15:12.013 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-04-23T03:15:12.013 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-04-23T03:15:12.018 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-04-23T03:15:12.019 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-04-23T03:15:12.156 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-04-23T03:15:12.234 INFO:teuthology.orchestra.run.smithi007.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T03:15:12.234 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T03:15:12.909 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T03:15:12.910 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T03:15:13.263 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - AppStream 894 kB/s | 19 MB 00:21 2024-04-23T03:15:13.609 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T03:15:13.609 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:15:13.610 INFO:teuthology.orchestra.run.smithi007.stdout:Reinstalled: 2024-04-23T03:15:13.610 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T03:15:13.610 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:15:13.610 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-04-23T03:15:13.766 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T03:15:13.814 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T03:15:13.814 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T03:15:13.814 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T03:15:13.814 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-23T03:15:13.814 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T03:15:13.848 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-04-23T03:15:13.873 DEBUG:teuthology.parallel:result is None 2024-04-23T03:15:17.580 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-23T03:15:19.197 INFO:teuthology.orchestra.run.smithi149.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 16 kB 00:00 2024-04-23T03:15:20.582 INFO:teuthology.orchestra.run.smithi149.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-04-23T03:15:25.632 INFO:teuthology.orchestra.run.smithi149.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-23T03:15:27.646 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-23T03:15:27.646 INFO:teuthology.orchestra.run.smithi149.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-23T03:15:27.737 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-23T03:15:27.738 INFO:teuthology.orchestra.run.smithi149.stdout:Nothing to do. 2024-04-23T03:15:27.738 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-23T03:15:27.842 DEBUG:teuthology.orchestra.run.smithi149:> echo no | sudo yum reinstall kernel || true 2024-04-23T03:15:28.348 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:03 ago on Tue 23 Apr 2024 03:15:25 AM UTC. 2024-04-23T03:15:28.467 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-23T03:15:28.482 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-23T03:15:28.482 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-04-23T03:15:28.482 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-23T03:15:28.482 INFO:teuthology.orchestra.run.smithi149.stdout:Reinstalling: 2024-04-23T03:15:28.483 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T03:15:28.483 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:15:28.483 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-23T03:15:28.483 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-23T03:15:28.483 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:15:28.483 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 255 k 2024-04-23T03:15:28.484 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 0 2024-04-23T03:15:28.484 INFO:teuthology.orchestra.run.smithi149.stderr:Operation aborted. 2024-04-23T03:15:28.544 INFO:teuthology.orchestra.run.smithi149.stdout:Is this ok [y/N]: 2024-04-23T03:15:28.544 DEBUG:teuthology.orchestra.run.smithi149:> sudo yum reinstall -y kernel || true 2024-04-23T03:15:29.044 INFO:teuthology.orchestra.run.smithi149.stdout:Last metadata expiration check: 0:00:04 ago on Tue 23 Apr 2024 03:15:25 AM UTC. 2024-04-23T03:15:29.161 INFO:teuthology.orchestra.run.smithi149.stdout:Dependencies resolved. 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout: Package Architecture Version Repository Size 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:Reinstalling: 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction Summary 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:================================================================================ 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:Total download size: 255 k 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:Installed size: 0 2024-04-23T03:15:29.162 INFO:teuthology.orchestra.run.smithi149.stdout:Downloading Packages: 2024-04-23T03:15:30.175 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-437.el9.x86_64.rpm 352 kB/s | 255 kB 00:00 2024-04-23T03:15:30.176 INFO:teuthology.orchestra.run.smithi149.stdout:-------------------------------------------------------------------------------- 2024-04-23T03:15:30.176 INFO:teuthology.orchestra.run.smithi149.stdout:Total 251 kB/s | 255 kB 00:01 2024-04-23T03:15:30.197 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction check 2024-04-23T03:15:30.396 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction check succeeded. 2024-04-23T03:15:30.396 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction test 2024-04-23T03:15:30.400 INFO:teuthology.orchestra.run.smithi149.stdout:Transaction test succeeded. 2024-04-23T03:15:30.401 INFO:teuthology.orchestra.run.smithi149.stdout:Running transaction 2024-04-23T03:15:30.523 INFO:teuthology.orchestra.run.smithi149.stdout: Preparing : 1/1 2024-04-23T03:15:30.601 INFO:teuthology.orchestra.run.smithi149.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T03:15:30.601 INFO:teuthology.orchestra.run.smithi149.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T03:15:31.204 INFO:teuthology.orchestra.run.smithi149.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T03:15:31.204 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-23T03:15:31.869 INFO:teuthology.orchestra.run.smithi149.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-23T03:15:31.869 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:15:31.869 INFO:teuthology.orchestra.run.smithi149.stdout:Reinstalled: 2024-04-23T03:15:31.869 INFO:teuthology.orchestra.run.smithi149.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-23T03:15:31.869 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:15:31.869 INFO:teuthology.orchestra.run.smithi149.stdout:Complete! 2024-04-23T03:15:32.062 DEBUG:teuthology.orchestra.run.smithi149:> rpm -q kernel | sort -rV | head -n 1 2024-04-23T03:15:32.109 INFO:teuthology.orchestra.run.smithi149.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-23T03:15:32.109 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-23T03:15:32.109 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-23T03:15:32.109 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-23T03:15:32.109 DEBUG:teuthology.orchestra.run.smithi149:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-23T03:15:32.135 INFO:teuthology.orchestra.run.smithi149.stdout:ttyS1 2024-04-23T03:15:32.152 DEBUG:teuthology.parallel:result is None 2024-04-23T03:15:32.152 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-23T03:15:32.164 INFO:teuthology.task.internal:Creating test directory... 2024-04-23T03:15:32.164 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T03:15:32.167 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-23T03:15:32.206 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-23T03:15:32.214 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-23T03:15:32.228 INFO:teuthology.task.internal:Creating archive directory... 2024-04-23T03:15:32.229 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T03:15:32.231 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-23T03:15:32.275 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-23T03:15:32.287 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-23T03:15:32.287 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T03:15:32.298 DEBUG:teuthology.orchestra.run.smithi149:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-23T03:15:32.331 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T03:15:32.342 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T03:15:32.343 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T03:15:32.352 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-23T03:15:32.354 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-23T03:15:32.365 INFO:teuthology.task.internal:Configuring sudo... 2024-04-23T03:15:32.365 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T03:15:32.387 DEBUG:teuthology.orchestra.run.smithi149:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-23T03:15:32.418 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-23T03:15:32.427 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-23T03:15:32.428 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T03:15:32.455 DEBUG:teuthology.orchestra.run.smithi149:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-23T03:15:32.472 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T03:15:32.558 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T03:15:32.603 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T03:15:32.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T03:15:32.728 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:15:32.728 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T03:15:32.794 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T03:15:32.829 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-23T03:15:32.887 DEBUG:teuthology.orchestra.run.smithi149:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T03:15:32.971 DEBUG:teuthology.orchestra.run.smithi149:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-23T03:15:33.010 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:15:33.010 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-23T03:15:33.074 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-04-23T03:15:33.076 DEBUG:teuthology.orchestra.run.smithi149:> sudo service rsyslog restart 2024-04-23T03:15:33.114 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T03:15:33.150 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T03:15:33.624 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-23T03:15:33.631 INFO:teuthology.task.internal:Starting timer... 2024-04-23T03:15:33.631 INFO:teuthology.run_tasks:Running task pcp... 2024-04-23T03:15:33.651 INFO:teuthology.run_tasks:Running task selinux... 2024-04-23T03:15:33.665 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-04-23T03:15:33.711 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-04-23T03:15:33.713 DEBUG:teuthology.orchestra.run.smithi149:> sudo service auditd rotate 2024-04-23T03:15:33.758 INFO:teuthology.orchestra.run.smithi149.stdout:Rotating logs: 2024-04-23T03:15:33.762 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-23T03:15:33.762 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-04-23T03:15:33.784 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-04-23T03:15:33.785 DEBUG:teuthology.orchestra.run.smithi149:> /usr/sbin/getenforce 2024-04-23T03:15:33.828 INFO:teuthology.orchestra.run.smithi149.stdout:Permissive 2024-04-23T03:15:33.828 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi149.front.sepia.ceph.com': 'permissive'} 2024-04-23T03:15:33.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-23T03:15:33.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:15:33.855 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-23T03:15:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:15:33.892 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-23T03:15:33.892 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-04-23T03:15:33.922 DEBUG:teuthology.orchestra.run.smithi149:> sudo /usr/sbin/setenforce permissive 2024-04-23T03:15:33.956 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-23T03:15:33.966 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-04-23T03:15:33.975 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-04-23T03:15:34.105 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-23T03:15:34.121 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-23T03:15:34.122 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi149.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-23T03:19:03.959 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi149.front.sepia.ceph.com')] 2024-04-23T03:19:03.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-23T03:19:03.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T03:19:04.039 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-04-23T03:19:04.125 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-23T03:19:04.125 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-23T03:19:04.126 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi149.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-23T03:19:04.200 DEBUG:teuthology.orchestra.run.smithi149:> true 2024-04-23T03:19:04.289 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi149.front.sepia.ceph.com' 2024-04-23T03:19:04.290 INFO:teuthology.run_tasks:Running task clock... 2024-04-23T03:19:04.300 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-23T03:19:04.300 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T03:19:04.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T03:19:04.303 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-23T03:19:04.303 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T03:19:04.330 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T03:19:04.343 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T03:19:04.358 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-23T03:19:04.367 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-04-23T03:19:04.371 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-23T03:19:04.377 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-04-23T03:19:04.391 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T03:19:04.404 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T03:19:04.407 INFO:teuthology.orchestra.run.smithi149.stderr:sudo: ntpd: command not found 2024-04-23T03:19:04.418 INFO:teuthology.orchestra.run.smithi149.stdout:506 Cannot talk to daemon 2024-04-23T03:19:04.432 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-23T03:19:04.445 INFO:teuthology.orchestra.run.smithi149.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-23T03:19:04.462 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-04-23T03:19:04.465 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T03:19:04.465 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-04-23T03:19:04.465 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.466 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.466 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.466 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.504 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-04-23T03:19:04.507 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T03:19:04.507 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-04-23T03:19:04.507 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.507 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.508 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.508 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:19:04.508 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-23T03:19:04.612 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '56e81df3aeb98b717efe2ab0537a8c60249f95f7'} 2024-04-23T03:19:04.612 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-23T03:19:04.613 INFO:tasks.cephadm:Cluster fsid is 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:04.613 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-23T03:19:04.613 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.c': '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]', 'mon.b': '172.21.15.149'} 2024-04-23T03:19:04.613 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-04-23T03:19:04.613 INFO:tasks.cephadm:First mgr is y 2024-04-23T03:19:04.613 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-23T03:19:04.613 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-04-23T03:19:04.639 DEBUG:teuthology.orchestra.run.smithi149:> sudo hostname $(hostname -s) 2024-04-23T03:19:04.663 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-23T03:19:04.664 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T03:19:04.904 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 23 03:19 /home/ubuntu/cephtest/cephadm 2024-04-23T03:19:04.904 DEBUG:teuthology.orchestra.run.smithi149:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-23T03:19:05.073 INFO:teuthology.orchestra.run.smithi149.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 23 03:19 /home/ubuntu/cephtest/cephadm 2024-04-23T03:19:05.074 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T03:19:05.091 DEBUG:teuthology.orchestra.run.smithi149:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-23T03:19:05.116 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-23T03:19:05.117 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-23T03:19:05.134 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-23T03:19:06.643 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-23T03:19:06.730 INFO:teuthology.orchestra.run.smithi149.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-23T03:19:25.725 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-23T03:19:25.726 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:19:29.929 INFO:teuthology.orchestra.run.smithi149.stdout:{ 2024-04-23T03:19:29.929 INFO:teuthology.orchestra.run.smithi149.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-23T03:19:29.930 INFO:teuthology.orchestra.run.smithi149.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-23T03:19:29.930 INFO:teuthology.orchestra.run.smithi149.stdout: "repo_digests": [ 2024-04-23T03:19:29.930 INFO:teuthology.orchestra.run.smithi149.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-23T03:19:29.930 INFO:teuthology.orchestra.run.smithi149.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-23T03:19:29.930 INFO:teuthology.orchestra.run.smithi149.stdout: ] 2024-04-23T03:19:29.930 INFO:teuthology.orchestra.run.smithi149.stdout:} 2024-04-23T03:19:29.944 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-04-23T03:19:29.977 DEBUG:teuthology.orchestra.run.smithi149:> sudo mkdir -p /etc/ceph 2024-04-23T03:19:30.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-04-23T03:19:30.045 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 777 /etc/ceph 2024-04-23T03:19:30.068 INFO:tasks.cephadm:Writing seed config... 2024-04-23T03:19:30.069 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-23T03:19:30.069 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-23T03:19:30.069 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-23T03:19:30.070 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-23T03:19:30.070 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-23T03:19:30.070 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-23T03:19:30.070 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-23T03:19:30.070 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-23T03:19:30.070 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-23T03:19:30.071 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:19:30.071 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-23T03:19:30.104 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3d05926e-0120-11ef-bc93-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-23T03:19:30.104 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service 2024-04-23T03:19:30.148 DEBUG:teuthology.orchestra.run.smithi007:mgr.y> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service 2024-04-23T03:19:30.191 INFO:tasks.cephadm:Bootstrapping... 2024-04-23T03:19:30.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 3d05926e-0120-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:19:30.393 INFO:teuthology.orchestra.run.smithi007.stderr:-------------------------------------------------------------------------------- 2024-04-23T03:19:30.394 INFO:teuthology.orchestra.run.smithi007.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '3d05926e-0120-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-04-23T03:19:30.425 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: 4.9.4-dev 2024-04-23T03:19:30.427 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying podman|docker is present... 2024-04-23T03:19:30.449 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: 4.9.4-dev 2024-04-23T03:19:30.452 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying lvm2 is present... 2024-04-23T03:19:30.452 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying time synchronization is in place... 2024-04-23T03:19:30.459 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T03:19:30.465 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-23T03:19:30.472 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: enabled 2024-04-23T03:19:30.478 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: active 2024-04-23T03:19:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:Unit chronyd.service is enabled and running 2024-04-23T03:19:30.479 INFO:teuthology.orchestra.run.smithi007.stderr:Repeating the final host check... 2024-04-23T03:19:30.499 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: 4.9.4-dev 2024-04-23T03:19:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-23T03:19:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl is present 2024-04-23T03:19:30.502 INFO:teuthology.orchestra.run.smithi007.stderr:lvcreate is present 2024-04-23T03:19:30.508 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-23T03:19:30.514 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-23T03:19:30.520 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: enabled 2024-04-23T03:19:30.526 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: active 2024-04-23T03:19:30.527 INFO:teuthology.orchestra.run.smithi007.stderr:Unit chronyd.service is enabled and running 2024-04-23T03:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:Host looks OK 2024-04-23T03:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:Cluster fsid: 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:Acquiring lock 140432048500592 on /run/cephadm/3d05926e-0120-11ef-bc93-c7b262605968.lock 2024-04-23T03:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:Lock 140432048500592 acquired on /run/cephadm/3d05926e-0120-11ef-bc93-c7b262605968.lock 2024-04-23T03:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying IP 172.21.15.7 port 3300 ... 2024-04-23T03:19:30.528 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying IP 172.21.15.7 port 6789 ... 2024-04-23T03:19:30.529 INFO:teuthology.orchestra.run.smithi007.stderr:Base mon IP is 172.21.15.7, final addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-04-23T03:19:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-04-23T03:19:30.531 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-04-23T03:19:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-23T03:19:30.533 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-23T03:19:30.534 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-23T03:19:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-23T03:19:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-23T03:19:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-23T03:19:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-23T03:19:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-04-23T03:19:30.536 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-23T03:19:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-04-23T03:19:30.537 INFO:teuthology.orchestra.run.smithi007.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-23T03:19:30.538 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-23T03:19:30.643 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-23T03:19:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Getting image source signatures 2024-04-23T03:19:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-23T03:19:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-23T03:19:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-23T03:19:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-23T03:19:31.022 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-23T03:19:31.023 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-23T03:19:31.178 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Writing manifest to image destination 2024-04-23T03:19:31.339 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-23T03:19:31.705 INFO:teuthology.orchestra.run.smithi007.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-23T03:19:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-23T03:19:32.188 INFO:teuthology.orchestra.run.smithi007.stderr:Extracting ceph user uid/gid from container image... 2024-04-23T03:19:32.871 INFO:teuthology.orchestra.run.smithi007.stderr:stat: 167 167 2024-04-23T03:19:33.345 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial keys... 2024-04-23T03:19:33.727 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQDFKCdmwWFYJhAA7cxLZy035inKdfN7F11FaA== 2024-04-23T03:19:34.607 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQDGKCdmaqSeIBAA9tymnp7bvDIUB9yrzaHKlQ== 2024-04-23T03:19:35.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQDHKCdmpbuBGBAA+Vmf8T/8ksnfsd+h07nKeQ== 2024-04-23T03:19:35.987 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial monmap... 2024-04-23T03:19:36.344 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T03:19:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-23T03:19:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:36.345 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T03:19:36.850 INFO:teuthology.orchestra.run.smithi007.stderr:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-23T03:19:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:setting min_mon_release = octopus 2024-04-23T03:19:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: set fsid to 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-23T03:19:36.851 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:19:36.851 INFO:teuthology.orchestra.run.smithi007.stderr:Creating mon... 2024-04-23T03:19:40.382 INFO:teuthology.orchestra.run.smithi007.stderr:create mon.a on 2024-04-23T03:19:40.853 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-23T03:19:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3d05926e-0120-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3d05926e-0120-11ef-bc93-c7b262605968.target. 2024-04-23T03:19:41.125 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-3d05926e-0120-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-3d05926e-0120-11ef-bc93-c7b262605968.target. 2024-04-23T03:19:41.651 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to reset failed state of unit ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-23T03:19:41.659 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3d05926e-0120-11ef-bc93-c7b262605968.target.wants/ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3d05926e-0120-11ef-bc93-c7b262605968@.service. 2024-04-23T03:19:42.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:42 smithi007 podman[21531]: 2024-04-23 03:19:42.127143266 +0000 UTC m=+0.019216765 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-23T03:19:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:42 smithi007 podman[21531]: 2024-04-23 03:19:42.943641642 +0000 UTC m=+0.835715134 container create 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, version=8, io.openshift.tags=base centos centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, maintainer=Guillaume Abrioux ) 2024-04-23T03:19:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:44 smithi007 podman[21531]: 2024-04-23 03:19:44.575599314 +0000 UTC m=+2.467672804 container init 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, version=8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, ceph=True, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public) 2024-04-23T03:19:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:44 smithi007 podman[21531]: 2024-04-23 03:19:44.579778001 +0000 UTC m=+2.471851489 container start 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, CEPH_POINT_RELEASE=-17.2.0, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, name=centos-stream, ceph=True, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-23T03:19:44.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:44 smithi007 ceph-mon[21549]: mkfs 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:44.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:44 smithi007 ceph-mon[21549]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T03:19:45.511 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-23T03:19:45.518 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-23T03:19:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-23T03:19:45.519 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to enable service . firewalld.service is not available 2024-04-23T03:19:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mon to start... 2024-04-23T03:19:45.520 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mon... 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 bash[21531]: 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 systemd[1]: Started Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 ceph-mon[21549]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 ceph-mon[21549]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 ceph-mon[21549]: fsmap 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 ceph-mon[21549]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T03:19:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:45 smithi007 ceph-mon[21549]: mgrmap e1: no daemons active 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: cluster: 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: id: 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: services: 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 1.6637s) 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: data: 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-23T03:19:46.291 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-23T03:19:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: pgs: 2024-04-23T03:19:46.292 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:46.791 INFO:teuthology.orchestra.run.smithi007.stderr:mon is available 2024-04-23T03:19:46.792 INFO:teuthology.orchestra.run.smithi007.stderr:Assimilating anything we can from ceph.conf... 2024-04-23T03:19:46.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:46 smithi007 ceph-mon[21549]: from='client.? 172.21.15.7:0/245832353' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T03:19:47.295 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:47.295 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [global] 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: fsid = 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [mgr] 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [osd] 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-23T03:19:47.296 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-23T03:19:48.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:47 smithi007 ceph-mon[21549]: from='client.? 172.21.15.7:0/174394033' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T03:19:48.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:47 smithi007 ceph-mon[21549]: from='client.? 172.21.15.7:0/174394033' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T03:19:49.374 INFO:teuthology.orchestra.run.smithi007.stderr:Generating new minimal ceph.conf... 2024-04-23T03:19:50.346 INFO:teuthology.orchestra.run.smithi007.stderr:Restarting the monitor... 2024-04-23T03:19:50.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:50 smithi007 systemd[1]: Stopping Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:19:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:50 smithi007 bash[21766]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-23T03:19:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a[21545]: 2024-04-23T03:19:50.503+0000 7f58bcd87700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:19:50.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a[21545]: 2024-04-23T03:19:50.503+0000 7f58bcd87700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-23T03:19:51.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:50 smithi007 podman[21779]: 2024-04-23 03:19:50.729406419 +0000 UTC m=+0.322863338 container died 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public, architecture=x86_64) 2024-04-23T03:19:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 podman[21779]: 2024-04-23 03:19:53.181101882 +0000 UTC m=+2.774558843 container cleanup 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, version=8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, release=754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870) 2024-04-23T03:19:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 bash[21779]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a 2024-04-23T03:19:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 podman[21793]: 2024-04-23 03:19:53.340141604 +0000 UTC m=+2.604234073 container remove 7feb1aec6ca4332c4166a24017e27fd0d9bebb34f29944b96d3b64f0420c13af (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.buildah.version=1.19.8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, version=8, vendor=Red Hat, Inc., io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-23T03:19:53.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 bash[21804]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-23T03:19:53.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T03:19:53.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 systemd[1]: Stopped Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:19:53.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 systemd[1]: Starting Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 podman[21867]: 2024-04-23 03:19:53.535202761 +0000 UTC m=+0.019517455 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 podman[21867]: 2024-04-23 03:19:53.674344222 +0000 UTC m=+0.158658949 container create 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, io.openshift.expose-services=, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, release=754, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.openshift.tags=base centos centos-stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 podman[21867]: 2024-04-23 03:19:53.851380176 +0000 UTC m=+0.335694866 container init 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 podman[21867]: 2024-04-23 03:19:53.854481262 +0000 UTC m=+0.338795950 container start 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.buildah.version=1.19.8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.openshift.expose-services=, vcs-type=git) 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: pidfile_write: ignore empty --pid-file 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: load: jerasure load: lrc 2024-04-23T03:19:53.883 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: RocksDB version: 6.15.5 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Compile date Apr 18 2022 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: DB SUMMARY 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: DB Session ID: PRNSIB95E8289PK5CMO5 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: CURRENT file: CURRENT 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82028 ; 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.error_if_exists: 0 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.create_if_missing: 0 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.paranoid_checks: 1 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T03:19:53.884 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.env: 0x55b0d94c6860 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.fs: Posix File System 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.info_log: 0x55b0da1fbdc0 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.statistics: (nil) 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.use_fsync: 0 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_log_file_size: 0 2024-04-23T03:19:53.885 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.allow_fallocate: 1 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.use_direct_reads: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.db_log_dir: 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T03:19:53.887 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.write_buffer_manager: 0x55b0da2ec2a0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T03:19:53.888 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.unordered_write: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.row_cache: None 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.wal_filter: None 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.preserve_deletes: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.two_write_queues: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.atomic_flush: 0 2024-04-23T03:19:53.889 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.log_readahead_size: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_background_jobs: 2 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_background_compactions: -1 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_subcompactions: 1 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T03:19:53.890 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_open_files: -1 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_background_flushes: -1 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Compression algorithms supported: 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T03:19:53.891 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kZSTD supported: 0 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kXpressCompression supported: 0 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kLZ4Compression supported: 1 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kBZip2Compression supported: 0 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kZlibCompression supported: 1 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: kSnappyCompression supported: 1 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.merge_operator: 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_filter: None 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T03:19:53.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b0da1c7d00) 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: hash_index_allow_collision: 1 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 1 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-04-23T03:19:53.893 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x55b0da232170 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-04-23T03:19:53.894 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 4 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression: NoCompression 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.num_levels: 7 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T03:19:53.895 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T03:19:53.896 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T03:19:53.897 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.arena_block_size: 4194304 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T03:19:53.898 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.table_properties_collectors: 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.inplace_update_support: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.bloom_locality: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.max_successive_merges: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T03:19:53.899 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.ttl: 2592000 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.enable_blob_files: false 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.min_blob_size: 0 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842393881800, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-23T03:19:53.900 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842393882572, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 78344, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 76607, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10016, "raw_average_key_size": 47, "raw_value_size": 70916, "raw_average_value_size": 336, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713842393, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "c0d8d8fb-14f2-46ed-9337-9ccff6a2b7e0", "db_session_id": "PRNSIB95E8289PK5CMO5"}} 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842393882929, "job": 1, "event": "recovery_finished"} 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55b0da218a80 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: DB pointer 0x55b0da228000 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-23T03:19:53.901 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 78.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 78.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.902 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 23.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 23.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:19:53.903 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 78.27 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 78.27 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 148.2 0.00 0.00 1 0.001 0 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 23.36 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:19:53.904 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:19:53.905 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:53.905 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:19:53.905 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:53.951 INFO:teuthology.orchestra.run.smithi007.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???) e1 preinit fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).mds e1 new map 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).mds e1 print_map 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T03:19:54.161 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 ceph-mon[21886]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 bash[21867]: 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 2024-04-23T03:19:54.162 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:53 smithi007 systemd[1]: Started Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:19:54.945 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-23T03:19:54.945 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:19:54.946 INFO:teuthology.orchestra.run.smithi007.stderr:Creating mgr... 2024-04-23T03:19:54.946 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying port 9283 ... 2024-04-23T03:19:54.955 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:54 smithi007 ceph-mon[21886]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T03:19:54.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:54 smithi007 ceph-mon[21886]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2024-04-23T03:19:54.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:54 smithi007 ceph-mon[21886]: fsmap 2024-04-23T03:19:54.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:54 smithi007 ceph-mon[21886]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T03:19:54.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:54 smithi007 ceph-mon[21886]: mgrmap e1: no daemons active 2024-04-23T03:19:54.956 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:54 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2640609505' entity='client.admin' 2024-04-23T03:19:55.125 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to reset failed state of unit ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-23T03:19:55.133 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph-3d05926e-0120-11ef-bc93-c7b262605968.target.wants/ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-3d05926e-0120-11ef-bc93-c7b262605968@.service. 2024-04-23T03:19:55.559 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 podman[22105]: 2024-04-23 03:19:55.416956481 +0000 UTC m=+0.018227420 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-23T03:19:55.559 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 podman[22105]: 2024-04-23 03:19:55.558342084 +0000 UTC m=+0.159613010 container create 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, architecture=x86_64, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, version=8, GIT_BRANCH=HEAD, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-23T03:19:55.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 podman[22105]: 2024-04-23 03:19:55.743670207 +0000 UTC m=+0.344941134 container init 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, release=754, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, architecture=x86_64, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git) 2024-04-23T03:19:55.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 podman[22105]: 2024-04-23 03:19:55.746396776 +0000 UTC m=+0.347667703 container start 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container, ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-23T03:19:55.849 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-23T03:19:55.856 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-23T03:19:55.857 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-23T03:19:55.857 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to enable service . firewalld.service is not available 2024-04-23T03:19:55.865 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-23T03:19:55.872 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-23T03:19:55.873 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-23T03:19:55.873 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-23T03:19:55.873 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr to start... 2024-04-23T03:19:55.874 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr... 2024-04-23T03:19:56.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 bash[22105]: 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf 2024-04-23T03:19:56.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 systemd[1]: Started Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:19:56.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:55.895+0000 7f685b38e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:19:56.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:56.033+0000 7f685b38e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:19:56.138 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:56.135+0000 7f685b38e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "3d05926e-0120-11ef-bc93-c7b262605968", 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.440 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:56.441 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-23T03:19:56.442 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-23T03:19:56.442 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-23T03:19:56.442 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-23T03:19:56.445 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-23T03:19:56.445 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-23T03:19:56.445 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.445 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-23T03:19:56.445 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-23T03:19:56.445 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-23T03:19:56.446 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-23T03:19:44.625226+0000", 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-23T03:19:56.447 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:19:56.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:56 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/640764839' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:19:56.742 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:56.468+0000 7f685b38e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:19:56.742 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:56.555+0000 7f685b38e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:19:57.024 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (1/15)... 2024-04-23T03:19:58.019 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:57.763+0000 7f685b38e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:19:58.019 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:57.932+0000 7f685b38e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:19:58.019 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:58.017+0000 7f685b38e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:19:58.780 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:58.341+0000 7f685b38e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:19:58.780 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:58.434+0000 7f685b38e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:19:58.780 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:58.530+0000 7f685b38e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:19:59.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:58.778+0000 7f685b38e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:19:59.445 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:59 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:59.246+0000 7f685b38e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "3d05926e-0120-11ef-bc93-c7b262605968", 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-23T03:19:59.531 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-23T03:19:59.532 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-23T03:19:59.533 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:59.534 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-23T03:19:59.535 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-23T03:19:44.625226+0000", 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-23T03:19:59.536 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:00.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:19:59 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2777939644' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:20:00.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:59 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:59.758+0000 7f685b38e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:20:00.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:19:59 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:19:59.854+0000 7f685b38e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:20:00.168 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (2/15)... 2024-04-23T03:20:00.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:00.164+0000 7f685b38e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:20:00.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:00.258+0000 7f685b38e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:20:01.030 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:00.677+0000 7f685b38e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:20:01.030 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:00.815+0000 7f685b38e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:20:01.030 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:00.911+0000 7f685b38e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:20:01.030 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:01.028+0000 7f685b38e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:20:01.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:01.120+0000 7f685b38e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:20:02.066 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:01.692+0000 7f685b38e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:02.065+0000 7f685b38e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: Activating manager daemon y 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: mgrmap e2: y(active, starting, since 0.00271543s) 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:20:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: Manager daemon y is now available 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' 2024-04-23T03:20:02.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:02 smithi007 ceph-mon[21886]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "3d05926e-0120-11ef-bc93-c7b262605968", 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-23T03:20:02.755 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-23T03:20:02.756 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-23T03:20:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-23T03:20:02.757 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-23T03:20:02.760 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-23T03:20:02.761 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-23T03:19:44.625226+0000", 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-23T03:20:02.762 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:03.308 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (3/15)... 2024-04-23T03:20:03.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:03 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2832737290' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:20:03.343 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:03 smithi007 ceph-mon[21886]: mgrmap e3: y(active, since 1.00592s) 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "3d05926e-0120-11ef-bc93-c7b262605968", 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-23T03:20:05.949 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 12, 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:05.950 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-23T03:20:05.951 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-23T03:20:05.952 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-23T03:19:44.625226+0000", 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-23T03:20:05.953 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:06.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:05 smithi007 ceph-mon[21886]: mgrmap e4: y(active, since 2s) 2024-04-23T03:20:06.581 INFO:teuthology.orchestra.run.smithi007.stderr:mgr is available 2024-04-23T03:20:07.216 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:06 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/92853688' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [global] 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: fsid = 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-23T03:20:07.218 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:20:07.219 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [mgr] 2024-04-23T03:20:07.219 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-23T03:20:07.219 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-23T03:20:07.219 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [osd] 2024-04-23T03:20:07.219 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-23T03:20:07.219 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-23T03:20:07.796 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling cephadm module... 2024-04-23T03:20:08.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:07 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/479737360' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T03:20:08.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:07 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/479737360' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T03:20:09.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:08 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/577536685' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T03:20:09.203 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setuser ceph since I am not root 2024-04-23T03:20:09.203 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setgroup ceph since I am not root 2024-04-23T03:20:09.203 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:09.067+0000 7fb9b9ee8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:20:09.499 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:09.201+0000 7fb9b9ee8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:20:09.499 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:09.304+0000 7fb9b9ee8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:20:09.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:09.629+0000 7fb9b9ee8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:20:09.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:09.715+0000 7fb9b9ee8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:20:10.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/577536685' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T03:20:10.254 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:09 smithi007 ceph-mon[21886]: mgrmap e5: y(active, since 6s) 2024-04-23T03:20:10.255 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:20:10.255 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-23T03:20:10.255 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2024-04-23T03:20:10.255 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-23T03:20:10.255 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-23T03:20:10.255 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for the mgr to restart... 2024-04-23T03:20:10.841 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr epoch 5... 2024-04-23T03:20:11.089 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:10 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/252716389' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T03:20:11.089 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:10.915+0000 7fb9b9ee8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:20:11.503 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:11.087+0000 7fb9b9ee8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:20:11.503 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:11.174+0000 7fb9b9ee8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:20:11.504 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:11.501+0000 7fb9b9ee8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:20:11.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:11.595+0000 7fb9b9ee8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:20:11.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:11.691+0000 7fb9b9ee8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:20:12.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:11.940+0000 7fb9b9ee8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:20:12.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:12 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:12.401+0000 7fb9b9ee8000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:20:13.306 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:12 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:12.902+0000 7fb9b9ee8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:20:13.307 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:12.999+0000 7fb9b9ee8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:20:13.307 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:13.305+0000 7fb9b9ee8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:20:13.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:13.397+0000 7fb9b9ee8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:20:14.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:13.814+0000 7fb9b9ee8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:20:14.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:13.950+0000 7fb9b9ee8000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:20:14.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:14.046+0000 7fb9b9ee8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:20:14.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:14.165+0000 7fb9b9ee8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:20:14.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:14.257+0000 7fb9b9ee8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:20:15.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:14.830+0000 7fb9b9ee8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:20:15.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:15.202+0000 7fb9b9ee8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:20:16.267 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:20:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-23T03:20:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "initialized": true 2024-04-23T03:20:16.268 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: Active manager daemon y restarted 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: Activating manager daemon y 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: mgrmap e6: y(active, starting, since 0.0526917s) 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:20:16.577 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: Manager daemon y is now available 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:20:16.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:20:16.795 INFO:teuthology.orchestra.run.smithi007.stderr:mgr epoch 5 is available 2024-04-23T03:20:16.796 INFO:teuthology.orchestra.run.smithi007.stderr:Setting orchestrator backend to cephadm... 2024-04-23T03:20:17.221 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:20:16] ENGINE Bus STARTING 2024-04-23T03:20:17.221 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:17 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:20:17] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:20:17.222 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:17 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:20:17] ENGINE Bus STARTED 2024-04-23T03:20:17.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:17 smithi007 ceph-mon[21886]: mgrmap e7: y(active, since 1.05477s) 2024-04-23T03:20:17.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:17 smithi007 ceph-mon[21886]: [23/Apr/2024:03:20:16] ENGINE Bus STARTING 2024-04-23T03:20:17.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:17 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:17.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:17 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:18.598 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: value unchanged 2024-04-23T03:20:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:18 smithi007 ceph-mon[21886]: [23/Apr/2024:03:20:17] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:20:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:18 smithi007 ceph-mon[21886]: [23/Apr/2024:03:20:17] ENGINE Bus STARTED 2024-04-23T03:20:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:18 smithi007 ceph-mon[21886]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:18 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:18 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:18 smithi007 ceph-mon[21886]: mgrmap e8: y(active, since 2s) 2024-04-23T03:20:19.142 INFO:teuthology.orchestra.run.smithi007.stderr:Generating ssh key... 2024-04-23T03:20:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:19 smithi007 ceph-mon[21886]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:21.145 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-mon[21886]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:21.145 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-mon[21886]: Generating ssh key... 2024-04-23T03:20:21.145 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:21.146 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:21.146 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: Generating public/private rsa key pair. 2024-04-23T03:20:21.146 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: Your identification has been saved in /tmp/tmp68ky3txd/key. 2024-04-23T03:20:21.146 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: Your public key has been saved in /tmp/tmp68ky3txd/key.pub. 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: The key fingerprint is: 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: SHA256:HHMqtGR/lun/imq/2bJh2rixXeNL8FSd+xZPw7i77i4 ceph-3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: The key's randomart image is: 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: +---[RSA 3072]----+ 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | | 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | ..| 2024-04-23T03:20:21.147 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | + o . ...| 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | + + = o .o .| 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | o S * .. =.| 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | . + + ..=| 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | . + =. +| 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | .O.E ... | 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: | .*+O+@Xo | 2024-04-23T03:20:21.148 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: +----[SHA256]-----+ 2024-04-23T03:20:22.228 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYFoHdwC/GOydxXC7fbwotdx8113tV0HUtNz38cNxRgcy7W+JioHkrXeVhXBtotMk3zROJ1fpGDjt055oI9ockNzGgUV9PP7J0uJRc/yX9Jgj4OaSAMEbQBBH0HqowggeUIUFInX0W5QheqvVUIePhi8t6zuPaKnsrayu0+sar0X6WWIID+Bq3JKVd3Czp0fjpH/mh0Xq1fz/zpo3LNzwPH5PXVmxEYvjI9I8eXH9emOiVpRngC0alGTvpcY+wluVVp+S6vWi2CD8UFZ2Yd/ZXAxVhGiOS639a4zYXdQ2NmrkQSawC8lUwg2MDPwkr7pWgpz8+9Dvfr5tLLLyb3ux2pXB5dsGraZ5LMQgUrVOzLsk38Srd+BkoPNp0qMyZ8LKNdf+m6O6tbPhLu2Vew8eQy3g9VI6a9dqErRtoN6cQ/Pkc8asSvcEH2FggUhwWGfbwxfqejKNZAH23uozXcWsts7Xbtz7WIFACn0fqdxu5WhZeoYT/qM2w5z4UZsiCgpk= ceph-3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:20:22.780 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-23T03:20:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:Adding key to root@localhost authorized_keys... 2024-04-23T03:20:22.781 INFO:teuthology.orchestra.run.smithi007.stderr:Adding host smithi007... 2024-04-23T03:20:23.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:23 smithi007 ceph-mon[21886]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:24.279 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Added host 'smithi007' with addr '172.21.15.7' 2024-04-23T03:20:24.845 INFO:teuthology.orchestra.run.smithi007.stderr:Deploying unmanaged mon service... 2024-04-23T03:20:25.195 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:24 smithi007 ceph-mon[21886]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:25.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:24 smithi007 ceph-mon[21886]: Deploying cephadm binary to smithi007 2024-04-23T03:20:25.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:24 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:25.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:24 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:25.505 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-23T03:20:26.208 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:25 smithi007 ceph-mon[21886]: Added host smithi007 2024-04-23T03:20:26.208 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:25 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:26.210 INFO:teuthology.orchestra.run.smithi007.stderr:Deploying unmanaged mgr service... 2024-04-23T03:20:26.971 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-23T03:20:26.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:26 smithi007 ceph-mon[21886]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:26.996 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:26 smithi007 ceph-mon[21886]: Saving service mon spec with placement count:5 2024-04-23T03:20:28.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:27 smithi007 ceph-mon[21886]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:28.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:27 smithi007 ceph-mon[21886]: Saving service mgr spec with placement count:2 2024-04-23T03:20:28.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:27 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:29 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1386498302' entity='client.admin' 2024-04-23T03:20:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:29 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:29 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:31.462 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling the dashboard module... 2024-04-23T03:20:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:31 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1993639646' entity='client.admin' 2024-04-23T03:20:31.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:31 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:33.211 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:32 smithi007 ceph-mon[21886]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:20:33.211 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:32 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1622052594' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T03:20:33.212 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:32 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setuser ceph since I am not root 2024-04-23T03:20:33.212 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:32 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setgroup ceph since I am not root 2024-04-23T03:20:33.212 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:33.110+0000 7ff36f382000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:20:33.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:33.245+0000 7ff36f382000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:20:33.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:33.347+0000 7ff36f382000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:20:33.942 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:33.666+0000 7ff36f382000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:20:33.942 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:33.752+0000 7ff36f382000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:20:34.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1622052594' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T03:20:34.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:33 smithi007 ceph-mon[21886]: mgrmap e9: y(active, since 17s) 2024-04-23T03:20:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:20:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-23T03:20:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2024-04-23T03:20:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-23T03:20:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-23T03:20:34.561 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:35.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:34 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2223237645' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T03:20:35.182 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:34.969+0000 7ff36f382000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:20:35.182 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:35 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:35.143+0000 7ff36f382000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:20:35.185 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for the mgr to restart... 2024-04-23T03:20:35.185 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr epoch 9... 2024-04-23T03:20:35.451 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:35 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:35.242+0000 7ff36f382000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:20:35.771 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:35 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:35.575+0000 7ff36f382000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:20:35.771 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:35 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:35.673+0000 7ff36f382000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:20:35.771 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:35 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:35.769+0000 7ff36f382000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:20:36.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:36.017+0000 7ff36f382000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:20:36.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:36.476+0000 7ff36f382000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:20:37.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:36.977+0000 7ff36f382000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:20:37.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:37.074+0000 7ff36f382000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:20:37.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:37.379+0000 7ff36f382000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:20:37.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:37.472+0000 7ff36f382000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:20:38.248 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:37.896+0000 7ff36f382000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:20:38.248 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:38.033+0000 7ff36f382000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:20:38.248 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:38.128+0000 7ff36f382000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:20:38.248 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:38.246+0000 7ff36f382000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:20:38.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:38.338+0000 7ff36f382000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:20:39.280 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:38.907+0000 7ff36f382000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:20:39.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:39 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:20:39.278+0000 7ff36f382000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: Active manager daemon y restarted 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: Activating manager daemon y 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: mgrmap e10: y(active, starting, since 0.616749s) 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:20:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:20:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: Manager daemon y is now available 2024-04-23T03:20:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:20:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:20:40.907 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:20:40] ENGINE Bus STARTING 2024-04-23T03:20:40.907 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:20:40] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:20:40.907 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:20:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:20:40] ENGINE Bus STARTED 2024-04-23T03:20:40.907 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-23T03:20:40.907 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-23T03:20:40.908 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "initialized": true 2024-04-23T03:20:40.908 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-23T03:20:41.678 INFO:teuthology.orchestra.run.smithi007.stderr:mgr epoch 9 is available 2024-04-23T03:20:41.678 INFO:teuthology.orchestra.run.smithi007.stderr:Generating a dashboard self-signed certificate... 2024-04-23T03:20:41.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: [23/Apr/2024:03:20:40] ENGINE Bus STARTING 2024-04-23T03:20:41.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: [23/Apr/2024:03:20:40] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:20:41.974 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: [23/Apr/2024:03:20:40] ENGINE Bus STARTED 2024-04-23T03:20:41.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:41.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T03:20:41.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: mgrmap e11: y(active, since 1.61868s) 2024-04-23T03:20:41.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T03:20:41.975 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:41 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:42.614 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-23T03:20:43.163 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial admin user... 2024-04-23T03:20:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:43 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:43 smithi007 ceph-mon[21886]: mgrmap e12: y(active, since 2s) 2024-04-23T03:20:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:43 smithi007 ceph-mon[21886]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:43 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:43.481 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:43 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:44.049 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$2Ng1/riaoNQqrYMD77jdm.aek9rpKOUNIL.TRw/Ameq1rFZb9FaSe", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713842444, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-23T03:20:45.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:44 smithi007 ceph-mon[21886]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:45.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:45.242 INFO:teuthology.orchestra.run.smithi007.stderr:Fetching dashboard port number... 2024-04-23T03:20:45.923 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 8443 2024-04-23T03:20:46.568 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-23T03:20:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-23T03:20:46.576 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-23T03:20:46.577 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-23T03:20:46.578 INFO:teuthology.orchestra.run.smithi007.stderr:Ceph Dashboard is now available at: 2024-04-23T03:20:46.579 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:46.579 INFO:teuthology.orchestra.run.smithi007.stderr: URL: https://smithi007:8443/ 2024-04-23T03:20:46.579 INFO:teuthology.orchestra.run.smithi007.stderr: User: admin 2024-04-23T03:20:46.579 INFO:teuthology.orchestra.run.smithi007.stderr: Password: 2qktyzwhx0 2024-04-23T03:20:46.579 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:46.579 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling autotune for osd_memory_target 2024-04-23T03:20:47.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:46 smithi007 ceph-mon[21886]: mgrmap e13: y(active, since 6s) 2024-04-23T03:20:47.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:46 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/683207369' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T03:20:48.653 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-23T03:20:49.363 INFO:teuthology.orchestra.run.smithi007.stderr:You can access the Ceph CLI with: 2024-04-23T03:20:49.363 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:49.363 INFO:teuthology.orchestra.run.smithi007.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: ceph telemetry on 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:For more information see: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:20:49.364 INFO:teuthology.orchestra.run.smithi007.stderr:Bootstrap complete. 2024-04-23T03:20:49.409 INFO:tasks.cephadm:Fetching config... 2024-04-23T03:20:49.409 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:20:49.409 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-23T03:20:49.428 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-23T03:20:49.428 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:20:49.429 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-23T03:20:49.485 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-23T03:20:49.485 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:20:49.485 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-23T03:20:49.553 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-23T03:20:49.553 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:20:49.553 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-23T03:20:49.610 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-23T03:20:49.610 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYFoHdwC/GOydxXC7fbwotdx8113tV0HUtNz38cNxRgcy7W+JioHkrXeVhXBtotMk3zROJ1fpGDjt055oI9ockNzGgUV9PP7J0uJRc/yX9Jgj4OaSAMEbQBBH0HqowggeUIUFInX0W5QheqvVUIePhi8t6zuPaKnsrayu0+sar0X6WWIID+Bq3JKVd3Czp0fjpH/mh0Xq1fz/zpo3LNzwPH5PXVmxEYvjI9I8eXH9emOiVpRngC0alGTvpcY+wluVVp+S6vWi2CD8UFZ2Yd/ZXAxVhGiOS639a4zYXdQ2NmrkQSawC8lUwg2MDPwkr7pWgpz8+9Dvfr5tLLLyb3ux2pXB5dsGraZ5LMQgUrVOzLsk38Srd+BkoPNp0qMyZ8LKNdf+m6O6tbPhLu2Vew8eQy3g9VI6a9dqErRtoN6cQ/Pkc8asSvcEH2FggUhwWGfbwxfqejKNZAH23uozXcWsts7Xbtz7WIFACn0fqdxu5WhZeoYT/qM2w5z4UZsiCgpk= ceph-3d05926e-0120-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T03:20:49.687 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYFoHdwC/GOydxXC7fbwotdx8113tV0HUtNz38cNxRgcy7W+JioHkrXeVhXBtotMk3zROJ1fpGDjt055oI9ockNzGgUV9PP7J0uJRc/yX9Jgj4OaSAMEbQBBH0HqowggeUIUFInX0W5QheqvVUIePhi8t6zuPaKnsrayu0+sar0X6WWIID+Bq3JKVd3Czp0fjpH/mh0Xq1fz/zpo3LNzwPH5PXVmxEYvjI9I8eXH9emOiVpRngC0alGTvpcY+wluVVp+S6vWi2CD8UFZ2Yd/ZXAxVhGiOS639a4zYXdQ2NmrkQSawC8lUwg2MDPwkr7pWgpz8+9Dvfr5tLLLyb3ux2pXB5dsGraZ5LMQgUrVOzLsk38Srd+BkoPNp0qMyZ8LKNdf+m6O6tbPhLu2Vew8eQy3g9VI6a9dqErRtoN6cQ/Pkc8asSvcEH2FggUhwWGfbwxfqejKNZAH23uozXcWsts7Xbtz7WIFACn0fqdxu5WhZeoYT/qM2w5z4UZsiCgpk= ceph-3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:20:49.698 DEBUG:teuthology.orchestra.run.smithi149:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYFoHdwC/GOydxXC7fbwotdx8113tV0HUtNz38cNxRgcy7W+JioHkrXeVhXBtotMk3zROJ1fpGDjt055oI9ockNzGgUV9PP7J0uJRc/yX9Jgj4OaSAMEbQBBH0HqowggeUIUFInX0W5QheqvVUIePhi8t6zuPaKnsrayu0+sar0X6WWIID+Bq3JKVd3Czp0fjpH/mh0Xq1fz/zpo3LNzwPH5PXVmxEYvjI9I8eXH9emOiVpRngC0alGTvpcY+wluVVp+S6vWi2CD8UFZ2Yd/ZXAxVhGiOS639a4zYXdQ2NmrkQSawC8lUwg2MDPwkr7pWgpz8+9Dvfr5tLLLyb3ux2pXB5dsGraZ5LMQgUrVOzLsk38Srd+BkoPNp0qMyZ8LKNdf+m6O6tbPhLu2Vew8eQy3g9VI6a9dqErRtoN6cQ/Pkc8asSvcEH2FggUhwWGfbwxfqejKNZAH23uozXcWsts7Xbtz7WIFACn0fqdxu5WhZeoYT/qM2w5z4UZsiCgpk= ceph-3d05926e-0120-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-23T03:20:49.739 INFO:teuthology.orchestra.run.smithi149.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCYFoHdwC/GOydxXC7fbwotdx8113tV0HUtNz38cNxRgcy7W+JioHkrXeVhXBtotMk3zROJ1fpGDjt055oI9ockNzGgUV9PP7J0uJRc/yX9Jgj4OaSAMEbQBBH0HqowggeUIUFInX0W5QheqvVUIePhi8t6zuPaKnsrayu0+sar0X6WWIID+Bq3JKVd3Czp0fjpH/mh0Xq1fz/zpo3LNzwPH5PXVmxEYvjI9I8eXH9emOiVpRngC0alGTvpcY+wluVVp+S6vWi2CD8UFZ2Yd/ZXAxVhGiOS639a4zYXdQ2NmrkQSawC8lUwg2MDPwkr7pWgpz8+9Dvfr5tLLLyb3ux2pXB5dsGraZ5LMQgUrVOzLsk38Srd+BkoPNp0qMyZ8LKNdf+m6O6tbPhLu2Vew8eQy3g9VI6a9dqErRtoN6cQ/Pkc8asSvcEH2FggUhwWGfbwxfqejKNZAH23uozXcWsts7Xbtz7WIFACn0fqdxu5WhZeoYT/qM2w5z4UZsiCgpk= ceph-3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:20:49.750 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-23T03:20:50.059 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:49 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2822051217' entity='client.admin' 2024-04-23T03:20:53.363 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-23T03:20:53.363 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-23T03:20:53.678 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:53 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3866533549' entity='client.admin' 2024-04-23T03:20:56.604 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi149 2024-04-23T03:20:56.605 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:20:56.605 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.conf 2024-04-23T03:20:56.622 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:20:56.623 DEBUG:teuthology.orchestra.run.smithi149:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:20:56.680 INFO:tasks.cephadm:Adding host smithi149 to orchestrator... 2024-04-23T03:20:56.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch host add smithi149 2024-04-23T03:20:56.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:56.702 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:20:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:57 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:20:59.465 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi149' with addr '172.21.15.149' 2024-04-23T03:21:00.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:00.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: Deploying cephadm binary to smithi149 2024-04-23T03:21:00.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:00.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:00.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:00.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:00.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:20:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:00.070 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-23T03:21:00.967 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:00 smithi007 ceph-mon[21886]: Added host smithi149 2024-04-23T03:21:00.968 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:00 smithi007 ceph-mon[21886]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:00.968 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:00 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:02.026 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:21:02.026 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.149", "hostname": "smithi149", "labels": [], "status": ""}] 2024-04-23T03:21:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:02 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:02.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:02 smithi007 ceph-mon[21886]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:02.606 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-23T03:21:02.606 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-23T03:21:03.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:03 smithi007 ceph-mon[21886]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:21:05.131 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-04-23T03:21:05.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:04 smithi007 ceph-mon[21886]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:05.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:04 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:05.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:04 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:05.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:04 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:21:05.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:04 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:05.155 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:04 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2958156472' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T03:21:05.709 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-04-23T03:21:05.709 INFO:tasks.cephadm:Adding mon.c on smithi007 2024-04-23T03:21:05.709 INFO:tasks.cephadm:Adding mon.b on smithi149 2024-04-23T03:21:05.710 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi149:172.21.15.149=b' 2024-04-23T03:21:06.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:06 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2958156472' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T03:21:06.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:06 smithi007 ceph-mon[21886]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:06.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:06 smithi007 ceph-mon[21886]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:07.536 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled mon update... 2024-04-23T03:21:08.163 DEBUG:teuthology.orchestra.run.smithi007:mon.c> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.c.service 2024-04-23T03:21:08.165 DEBUG:teuthology.orchestra.run.smithi149:mon.b> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.b.service 2024-04-23T03:21:08.167 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T03:21:08.168 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T03:21:08.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi149:172.21.15.149=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:08.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi149:172.21.15.149=b;count:3 2024-04-23T03:21:08.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:08.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:08.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:08.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: Deploying daemon mon.c on smithi007 2024-04-23T03:21:08.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:08 smithi007 ceph-mon[21886]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:10.083 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:21:10.083 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":1,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","modified":"2024-04-23T03:19:36.264646Z","created":"2024-04-23T03:19:36.264646Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-23T03:21:10.085 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 1 2024-04-23T03:21:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:10 smithi007 podman[26664]: 2024-04-23 03:21:10.791959067 +0000 UTC m=+0.130019174 container create 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., vcs-type=git, release=754, architecture=x86_64) 2024-04-23T03:21:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:10 smithi007 ceph-mon[21886]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:10 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/2510465446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 podman[26664]: 2024-04-23 03:21:11.011298887 +0000 UTC m=+0.349358995 container init 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, io.openshift.tags=base centos centos-stream, release=754, architecture=x86_64, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vcs-type=git, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8) 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 podman[26664]: 2024-04-23 03:21:11.014594233 +0000 UTC m=+0.352654344 container start 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public) 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pidfile_write: ignore empty --pid-file 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: load: jerasure load: lrc 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: RocksDB version: 6.15.5 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Compile date Apr 18 2022 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: DB SUMMARY 2024-04-23T03:21:11.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: DB Session ID: G7BPTQNWJMXOKZK3240M 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: CURRENT file: CURRENT 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.error_if_exists: 0 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.create_if_missing: 0 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.paranoid_checks: 1 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.env: 0x5608b78c6860 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.fs: Posix File System 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.info_log: 0x5608b8d9fdc0 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T03:21:11.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.statistics: (nil) 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.use_fsync: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_log_file_size: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.allow_fallocate: 1 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.use_direct_reads: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.db_log_dir: 2024-04-23T03:21:11.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-23T03:21:11.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T03:21:11.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T03:21:11.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T03:21:11.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T03:21:11.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T03:21:11.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.write_buffer_manager: 0x5608b8e902d0 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T03:21:11.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.unordered_write: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.row_cache: None 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.wal_filter: None 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.preserve_deletes: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.two_write_queues: 0 2024-04-23T03:21:11.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.atomic_flush: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.log_readahead_size: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_background_jobs: 2 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_background_compactions: -1 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_subcompactions: 1 2024-04-23T03:21:11.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_open_files: -1 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_background_flushes: -1 2024-04-23T03:21:11.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Compression algorithms supported: 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kZSTD supported: 0 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kXpressCompression supported: 0 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kLZ4Compression supported: 1 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kBZip2Compression supported: 0 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kZlibCompression supported: 1 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: kSnappyCompression supported: 1 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T03:21:11.343 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.merge_operator: 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_filter: None 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5608b8d6bd00) 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_type: 0 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_index_type: 0 2024-04-23T03:21:11.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_shortening: 1 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: hash_index_allow_collision: 1 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: checksum: 1 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: no_block_cache: 0 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache: 0x5608b8dd6170 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_options: 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: capacity : 536870912 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_shard_bits : 4 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: strict_capacity_limit : 0 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-23T03:21:11.345 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_compressed: (nil) 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: persistent_cache: (nil) 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size: 4096 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size_deviation: 10 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_restart_interval: 16 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_block_restart_interval: 1 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: metadata_block_size: 4096 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: partition_filters: 0 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_delta_encoding: 1 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: whole_key_filtering: 1 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: verify_compression: 0 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: format_version: 4 2024-04-23T03:21:11.346 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_index_compression: 1 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_align: 0 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression: NoCompression 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.num_levels: 7 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T03:21:11.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T03:21:11.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.arena_block_size: 4194304 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T03:21:11.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.table_properties_collectors: 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.inplace_update_support: 0 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T03:21:11.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.bloom_locality: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.max_successive_merges: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.ttl: 2592000 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.enable_blob_files: false 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.min_blob_size: 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T03:21:11.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842471041163, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842471041471, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713842471, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "6e9b62cf-ad9a-450f-b548-18d9609356d0", "db_session_id": "G7BPTQNWJMXOKZK3240M"}} 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842471041797, "job": 1, "event": "recovery_finished"} 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5608b8dbca80 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: DB pointer 0x5608b8dcc000 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** DB Stats ** 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T03:21:11.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:11.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:11.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.9 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:11.355 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: using public_addrv [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: starting mon.c rank -1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(???) e0 preinit fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).mds e1 new map 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: e1 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: legacy client fscid: -1 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout: No filesystems configured 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T03:21:11.356 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mkfs 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: fsmap 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e1: no daemons active 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/245832353' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/174394033' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T03:21:11.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/174394033' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: fsmap 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: osdmap e1: 0 total, 0 up, 0 in 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e1: no daemons active 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2640609505' entity='client.admin' 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/640764839' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2777939644' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Activating manager daemon y 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e2: y(active, starting, since 0.00271543s) 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:21:11.358 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Manager daemon y is now available 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14100 172.21.15.7:0/3013938016' entity='mgr.y' 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2832737290' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e3: y(active, since 1.00592s) 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e4: y(active, since 2s) 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/92853688' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/479737360' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/479737360' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/577536685' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/577536685' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-23T03:21:11.359 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e5: y(active, since 6s) 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/252716389' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Active manager daemon y restarted 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Activating manager daemon y 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: osdmap e2: 0 total, 0 up, 0 in 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e6: y(active, starting, since 0.0526917s) 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Manager daemon y is now available 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.360 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e7: y(active, since 1.05477s) 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: [23/Apr/2024:03:20:16] ENGINE Bus STARTING 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: [23/Apr/2024:03:20:17] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: [23/Apr/2024:03:20:17] ENGINE Bus STARTED 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e8: y(active, since 2s) 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Generating ssh key... 2024-04-23T03:21:11.361 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Deploying cephadm binary to smithi007 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Added host smithi007 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.362 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Saving service mon spec with placement count:5 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Saving service mgr spec with placement count:2 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1386498302' entity='client.admin' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1993639646' entity='client.admin' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14120 172.21.15.7:0/938355301' entity='mgr.y' 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1622052594' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1622052594' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-23T03:21:11.363 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e9: y(active, since 17s) 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2223237645' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Active manager daemon y restarted 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Activating manager daemon y 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: osdmap e3: 0 total, 0 up, 0 in 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e10: y(active, starting, since 0.616749s) 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Manager daemon y is now available 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: [23/Apr/2024:03:20:40] ENGINE Bus STARTING 2024-04-23T03:21:11.364 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: [23/Apr/2024:03:20:40] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: [23/Apr/2024:03:20:40] ENGINE Bus STARTED 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e11: y(active, since 1.61868s) 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e12: y(active, since 2s) 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mgrmap e13: y(active, since 6s) 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/683207369' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2822051217' entity='client.admin' 2024-04-23T03:21:11.365 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3866533549' entity='client.admin' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi149", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.366 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Deploying cephadm binary to smithi149 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Added host smithi149 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi149", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:21:11.367 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2958156472' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2958156472' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi149:172.21.15.149=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi149:172.21.15.149=b;count:3 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:11.368 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: Deploying daemon mon.c on smithi007 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/2510465446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 ceph-mon[26683]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 bash[26664]: 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c 2024-04-23T03:21:11.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:11 smithi007 systemd[1]: Started Ceph mon.c for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:21:11.616 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T03:21:11.617 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: mon.a calling monitor election 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: mon.c calling monitor election 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:16.418 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: fsmap 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: mgrmap e13: y(active, since 36s) 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: overall HEALTH_OK 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:16.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: mon.a calling monitor election 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: mon.c calling monitor election 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: fsmap 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: mgrmap e13: y(active, since 36s) 2024-04-23T03:21:16.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: overall HEALTH_OK 2024-04-23T03:21:16.421 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:16.421 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:16.421 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:16.511 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:21:16.511 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":2,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","modified":"2024-04-23T03:21:11.060797Z","created":"2024-04-23T03:19:36.264646Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T03:21:16.513 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 2 2024-04-23T03:21:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:17 smithi007 ceph-mon[26683]: Deploying daemon mon.b on smithi149 2024-04-23T03:21:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:17 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/2862043966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:17 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:17 smithi007 ceph-mon[21886]: Deploying daemon mon.b on smithi149 2024-04-23T03:21:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:17 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/2862043966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:17 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:18.179 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T03:21:18.179 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T03:21:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:18 smithi007 ceph-mon[21886]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:18.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:18 smithi007 ceph-mon[26683]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:19.841 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:19 smithi149 systemd[1]: Starting Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:21:20.748 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:21:20.748 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":2,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","modified":"2024-04-23T03:21:11.060797Z","created":"2024-04-23T03:19:36.264646Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T03:21:20.750 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 2 2024-04-23T03:21:21.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:20 smithi149 podman[22684]: 2024-04-23 03:21:20.650873024 +0000 UTC m=+0.018232023 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-23T03:21:21.052 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:20 smithi149 podman[22684]: 2024-04-23 03:21:20.77797912 +0000 UTC m=+0.145338103 container create 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, ceph=True, RELEASE=HEAD, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, release=754) 2024-04-23T03:21:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:20 smithi007 ceph-mon[21886]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:20 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/4090272072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:20 smithi007 ceph-mon[26683]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:20 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/4090272072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 podman[22684]: 2024-04-23 03:21:21.147861021 +0000 UTC m=+0.515220004 container init 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, RELEASE=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., release=754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, name=centos-stream) 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 podman[22684]: 2024-04-23 03:21:21.151053999 +0000 UTC m=+0.518412983 container start 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, distribution-scope=public, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, name=centos-stream, vcs-type=git, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux ) 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: pidfile_write: ignore empty --pid-file 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: load: jerasure load: lrc 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: RocksDB version: 6.15.5 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Compile date Apr 18 2022 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: DB SUMMARY 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: DB Session ID: O6NLLIQK2S8WETGX5I9Z 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: CURRENT file: CURRENT 2024-04-23T03:21:21.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.error_if_exists: 0 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.create_if_missing: 0 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.paranoid_checks: 1 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.env: 0x556b27ec6860 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.fs: Posix File System 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.info_log: 0x556b2951bdc0 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.statistics: (nil) 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.use_fsync: 0 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_log_file_size: 0 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T03:21:21.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.allow_fallocate: 1 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.use_direct_reads: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.db_log_dir: 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T03:21:21.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.write_buffer_manager: 0x556b2960c240 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T03:21:21.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.unordered_write: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.row_cache: None 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.wal_filter: None 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.preserve_deletes: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.two_write_queues: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.atomic_flush: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T03:21:21.428 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.log_readahead_size: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_background_jobs: 2 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_background_compactions: -1 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_subcompactions: 1 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T03:21:21.429 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_open_files: -1 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_background_flushes: -1 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Compression algorithms supported: 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kZSTD supported: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kXpressCompression supported: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kLZ4Compression supported: 1 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kBZip2Compression supported: 0 2024-04-23T03:21:21.430 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kZlibCompression supported: 1 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: kSnappyCompression supported: 1 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.merge_operator: 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_filter: None 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556b294e7d00) 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-04-23T03:21:21.431 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_type: 0 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_index_type: 0 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_shortening: 1 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: hash_index_allow_collision: 1 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: checksum: 1 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: no_block_cache: 0 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache: 0x556b29552170 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_options: 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: capacity : 536870912 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_shard_bits : 4 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: strict_capacity_limit : 0 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_compressed: (nil) 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: persistent_cache: (nil) 2024-04-23T03:21:21.432 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size: 4096 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size_deviation: 10 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_restart_interval: 16 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_block_restart_interval: 1 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: metadata_block_size: 4096 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: partition_filters: 0 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_delta_encoding: 1 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: whole_key_filtering: 1 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: verify_compression: 0 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: format_version: 4 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_index_compression: 1 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_align: 0 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression: NoCompression 2024-04-23T03:21:21.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.num_levels: 7 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T03:21:21.434 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T03:21:21.435 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.arena_block_size: 4194304 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T03:21:21.436 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.table_properties_collectors: 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.inplace_update_support: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.bloom_locality: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.max_successive_merges: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.ttl: 2592000 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T03:21:21.437 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.enable_blob_files: false 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.min_blob_size: 0 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842481177119, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842481177422, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713842481, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "a7867ccb-6e6e-4bf4-bb06-8a91defbe9ce", "db_session_id": "O6NLLIQK2S8WETGX5I9Z"}} 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-23T03:21:21.438 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713842481177730, "job": 1, "event": "recovery_finished"} 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x556b29538a80 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: DB pointer 0x556b29548000 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** DB Stats ** 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:21.439 INFO:journalctl@ceph.mon.b.smithi149.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:21.440 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-23T03:21:21.441 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: using public_addr v2:172.21.15.149:0/0 -> [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: starting mon.b rank -1 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(???) e0 preinit fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).mds e1 new map 2024-04-23T03:21:21.442 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout: e1 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout: legacy client fscid: -1 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout: No filesystems configured 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-23T03:21:21.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:21.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/4090272072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:21.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-04-23T03:21:21.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 ceph-mon[22716]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-23T03:21:21.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 bash[22684]: 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d 2024-04-23T03:21:21.444 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:21 smithi149 systemd[1]: Started Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:21:22.732 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-23T03:21:22.733 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-23T03:21:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: mon.c calling monitor election 2024-04-23T03:21:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: mon.a calling monitor election 2024-04-23T03:21:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: fsmap 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: mgrmap e13: y(active, since 46s) 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-23T03:21:26.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: mon.c calling monitor election 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: mon.a calling monitor election 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:26.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: fsmap 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: mgrmap e13: y(active, since 46s) 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:26.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:26.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:26 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:27.558 INFO:teuthology.orchestra.run.smithi149.stdout: 2024-04-23T03:21:27.558 INFO:teuthology.orchestra.run.smithi149.stdout:{"epoch":3,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","modified":"2024-04-23T03:21:21.198553Z","created":"2024-04-23T03:19:36.264646Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:3300","nonce":0},{"type":"v1","addr":"172.21.15.149:6789","nonce":0}]},"addr":"172.21.15.149:6789/0","public_addr":"172.21.15.149:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-23T03:21:27.559 INFO:teuthology.orchestra.run.smithi149.stderr:dumped monmap epoch 3 2024-04-23T03:21:28.109 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-23T03:21:28.109 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-23T03:21:28.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:21:28.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: mon.c calling monitor election 2024-04-23T03:21:28.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: mon.a calling monitor election 2024-04-23T03:21:28.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:21:28.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: fsmap 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: mgrmap e13: y(active, since 46s) 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-23T03:21:28.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: mon.b (rank 2) addr [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] is down (out of quorum) 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T03:21:28.425 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:28.426 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:28.426 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:28.426 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: Reconfiguring daemon mon.a on smithi007 2024-04-23T03:21:28.426 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/4075014037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:28.426 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:28 smithi149 ceph-mon[22716]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:28.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: Reconfiguring daemon mon.a on smithi007 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/4075014037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:28.475 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[26683]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: Reconfiguring mon.a (unknown last config time)... 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:28.476 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:28.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: Reconfiguring daemon mon.a on smithi007 2024-04-23T03:21:28.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/4075014037' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-23T03:21:28.477 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:28 smithi007 ceph-mon[21886]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:29.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: mon.b calling monitor election 2024-04-23T03:21:29.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: mon.b calling monitor election 2024-04-23T03:21:29.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: mon.c calling monitor election 2024-04-23T03:21:29.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: mon.a calling monitor election 2024-04-23T03:21:29.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: fsmap 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: mgrmap e13: y(active, since 48s) 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: Cluster is now healthy 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[26683]: overall HEALTH_OK 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: mon.b calling monitor election 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: mon.b calling monitor election 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: mon.c calling monitor election 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: mon.a calling monitor election 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:21:29.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:29.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: fsmap 2024-04-23T03:21:29.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:29.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: mgrmap e13: y(active, since 48s) 2024-04-23T03:21:29.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-23T03:21:29.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: Cluster is now healthy 2024-04-23T03:21:29.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:29 smithi007 ceph-mon[21886]: overall HEALTH_OK 2024-04-23T03:21:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: mon.b calling monitor election 2024-04-23T03:21:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: mon.b calling monitor election 2024-04-23T03:21:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: mon.c calling monitor election 2024-04-23T03:21:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: mon.a calling monitor election 2024-04-23T03:21:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: fsmap 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: osdmap e4: 0 total, 0 up, 0 in 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: mgrmap e13: y(active, since 48s) 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: Cluster is now healthy 2024-04-23T03:21:29.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:29 smithi149 ceph-mon[22716]: overall HEALTH_OK 2024-04-23T03:21:30.026 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:30.026 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-04-23T03:21:30.026 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:21:30.026 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-04-23T03:21:30.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:30 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:30.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:30 smithi149 ceph-mon[22716]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:30.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:30 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:30.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:30 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2523354302' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[21886]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2523354302' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[26683]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:30.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:30.284 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:30 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2523354302' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:30.927 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-23T03:21:30.927 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:21:30.927 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T03:21:30.954 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:21:30.954 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:21:31.019 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:21:31.019 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.conf 2024-04-23T03:21:31.048 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:21:31.049 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:21:31.116 INFO:tasks.cephadm:Adding mgr.y on smithi007 2024-04-23T03:21:31.116 INFO:tasks.cephadm:Adding mgr.x on smithi149 2024-04-23T03:21:31.116 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi007=y;smithi149=x' 2024-04-23T03:21:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[26683]: Reconfiguring mon.c (monmap changed)... 2024-04-23T03:21:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[26683]: Reconfiguring daemon mon.c on smithi007 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[21886]: Reconfiguring mon.c (monmap changed)... 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:31 smithi007 ceph-mon[21886]: Reconfiguring daemon mon.c on smithi007 2024-04-23T03:21:31.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:31 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:31.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:31 smithi149 ceph-mon[22716]: Reconfiguring mon.c (monmap changed)... 2024-04-23T03:21:31.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:31 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:31.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:31 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:31.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:31 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:31.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:31 smithi149 ceph-mon[22716]: Reconfiguring daemon mon.c on smithi007 2024-04-23T03:21:32.664 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:32 smithi149 ceph-mon[22716]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:32.665 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:32 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:32.665 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:32 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:32.665 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:32 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:32.665 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:32 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[26683]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[21886]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:21:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:21:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:32 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:33.069 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled mgr update... 2024-04-23T03:21:33.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:33 smithi149 ceph-mon[22716]: Reconfiguring mon.b (monmap changed)... 2024-04-23T03:21:33.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:33 smithi149 ceph-mon[22716]: Reconfiguring daemon mon.b on smithi149 2024-04-23T03:21:33.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:33 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:33.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:33 smithi007 ceph-mon[21886]: Reconfiguring mon.b (monmap changed)... 2024-04-23T03:21:33.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:33 smithi007 ceph-mon[21886]: Reconfiguring daemon mon.b on smithi149 2024-04-23T03:21:33.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:33 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:33.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:33 smithi007 ceph-mon[26683]: Reconfiguring mon.b (monmap changed)... 2024-04-23T03:21:33.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:33 smithi007 ceph-mon[26683]: Reconfiguring daemon mon.b on smithi149 2024-04-23T03:21:33.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:33 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:34.102 DEBUG:teuthology.orchestra.run.smithi149:mgr.x> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service 2024-04-23T03:21:34.104 INFO:tasks.cephadm:Deploying OSDs... 2024-04-23T03:21:34.104 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:21:34.104 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T03:21:34.125 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T03:21:34.125 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-04-23T03:21:34.184 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T03:21:34.184 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:34.184 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-23T03:21:34.184 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:34.184 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:34.185 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-23 03:21:10.499526261 +0000 2024-04-23T03:21:34.185 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-23 03:18:45.845096656 +0000 2024-04-23T03:21:34.185 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-23 03:18:45.845096656 +0000 2024-04-23T03:21:34.185 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-23 03:18:45.845096656 +0000 2024-04-23T03:21:34.185 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T03:21:34.252 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-23T03:21:34.253 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-23T03:21:34.253 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00012312 s, 4.2 MB/s 2024-04-23T03:21:34.254 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T03:21:34.313 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-23 03:21:10.499526261 +0000 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-23 03:18:46.180091375 +0000 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-23 03:18:46.180091375 +0000 2024-04-23T03:21:34.372 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-23 03:18:46.180091375 +0000 2024-04-23T03:21:34.372 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T03:21:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-23T03:21:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-23T03:21:34.440 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000112252 s, 4.6 MB/s 2024-04-23T03:21:34.441 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T03:21:34.501 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-04-23T03:21:34.558 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T03:21:34.558 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:34.558 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-23T03:21:34.558 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:34.558 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:34.559 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-23 03:21:10.500526246 +0000 2024-04-23T03:21:34.559 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-23 03:18:46.505086253 +0000 2024-04-23T03:21:34.559 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-23 03:18:46.505086253 +0000 2024-04-23T03:21:34.559 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-23 03:18:46.505086253 +0000 2024-04-23T03:21:34.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T03:21:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-23T03:21:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-23T03:21:34.623 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000115977 s, 4.4 MB/s 2024-04-23T03:21:34.624 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T03:21:34.681 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-04-23T03:21:34.737 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T03:21:34.737 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-23 03:21:10.500526246 +0000 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-23 03:18:46.841080956 +0000 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-23 03:18:46.841080956 +0000 2024-04-23T03:21:34.738 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-23 03:18:46.841080956 +0000 2024-04-23T03:21:34.738 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T03:21:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-23T03:21:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-23T03:21:34.801 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.2633e-05 s, 5.5 MB/s 2024-04-23T03:21:34.803 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T03:21:34.859 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:21:34.860 DEBUG:teuthology.orchestra.run.smithi149:> dd if=/scratch_devs of=/dev/stdout 2024-04-23T03:21:34.875 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-23T03:21:34.875 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_1 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-23 03:21:19.688948506 +0000 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-23 03:18:42.494818316 +0000 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-23 03:18:42.494818316 +0000 2024-04-23T03:21:34.930 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-04-23 03:18:42.494818316 +0000 2024-04-23T03:21:34.931 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-23T03:21:34.991 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-23T03:21:34.991 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-23T03:21:34.991 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000208947 s, 2.5 MB/s 2024-04-23T03:21:34.993 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-23T03:21:35.048 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_2 2024-04-23T03:21:35.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi149=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:35.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: Saving service mgr spec with placement smithi007=y;smithi149=x;count:2 2024-04-23T03:21:35.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:35.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:35.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:35.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi149=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: Saving service mgr spec with placement smithi007=y;smithi149=x;count:2 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:35.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:35.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:35.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:21:35.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T03:21:35.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:21:35.085 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-23 03:21:19.688948506 +0000 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-23 03:18:42.831811206 +0000 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-23 03:18:42.831811206 +0000 2024-04-23T03:21:35.103 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-04-23 03:18:42.831811206 +0000 2024-04-23T03:21:35.104 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi149=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: Saving service mgr spec with placement smithi007=y;smithi149=x;count:2 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:21:35.164 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:35.166 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-23T03:21:35.166 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-23T03:21:35.166 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000193969 s, 2.6 MB/s 2024-04-23T03:21:35.167 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-23T03:21:35.222 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_3 2024-04-23T03:21:35.277 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-23 03:21:19.689948487 +0000 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-23 03:18:43.185803734 +0000 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-23 03:18:43.185803734 +0000 2024-04-23T03:21:35.278 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-04-23 03:18:43.185803734 +0000 2024-04-23T03:21:35.278 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-23T03:21:35.339 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-23T03:21:35.339 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-23T03:21:35.339 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000213474 s, 2.4 MB/s 2024-04-23T03:21:35.340 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-23T03:21:35.396 DEBUG:teuthology.orchestra.run.smithi149:> stat /dev/vg_nvme/lv_4 2024-04-23T03:21:35.450 INFO:teuthology.orchestra.run.smithi149.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-23T03:21:35.450 INFO:teuthology.orchestra.run.smithi149.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-23T03:21:35.450 INFO:teuthology.orchestra.run.smithi149.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-23T03:21:35.450 INFO:teuthology.orchestra.run.smithi149.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-23T03:21:35.450 INFO:teuthology.orchestra.run.smithi149.stdout:Context: system_u:object_r:device_t:s0 2024-04-23T03:21:35.450 INFO:teuthology.orchestra.run.smithi149.stdout:Access: 2024-04-23 03:21:19.689948487 +0000 2024-04-23T03:21:35.451 INFO:teuthology.orchestra.run.smithi149.stdout:Modify: 2024-04-23 03:18:43.525796556 +0000 2024-04-23T03:21:35.451 INFO:teuthology.orchestra.run.smithi149.stdout:Change: 2024-04-23 03:18:43.525796556 +0000 2024-04-23T03:21:35.451 INFO:teuthology.orchestra.run.smithi149.stdout: Birth: 2024-04-23 03:18:43.525796556 +0000 2024-04-23T03:21:35.451 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-23T03:21:35.512 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records in 2024-04-23T03:21:35.512 INFO:teuthology.orchestra.run.smithi149.stderr:1+0 records out 2024-04-23T03:21:35.512 INFO:teuthology.orchestra.run.smithi149.stderr:512 bytes copied, 0.000189425 s, 2.7 MB/s 2024-04-23T03:21:35.513 DEBUG:teuthology.orchestra.run.smithi149:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-23T03:21:35.568 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2024-04-23T03:21:35.568 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T03:21:36.139 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:35 smithi149 ceph-mon[22716]: Deploying daemon mgr.x on smithi149 2024-04-23T03:21:36.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:35 smithi007 ceph-mon[21886]: Deploying daemon mgr.x on smithi149 2024-04-23T03:21:36.189 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:35 smithi007 ceph-mon[26683]: Deploying daemon mgr.x on smithi149 2024-04-23T03:21:36.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 podman[23885]: 2024-04-23 03:21:36.356605109 +0000 UTC m=+0.116841792 container create acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, vcs-type=git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, ceph=True, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, version=8, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-23T03:21:36.723 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 podman[23885]: 2024-04-23 03:21:36.54201912 +0000 UTC m=+0.302255750 container init acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, ceph=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, release=754, io.openshift.expose-services=, vcs-type=git, com.redhat.component=centos-stream-container, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-23T03:21:36.723 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 podman[23885]: 2024-04-23 03:21:36.544524942 +0000 UTC m=+0.304761573 container start acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64) 2024-04-23T03:21:36.723 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-mgr[23904]: -- 172.21.15.149:0/315302085 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561e1d2b6340 con 0x561e1d28f000 2024-04-23T03:21:36.723 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 bash[23885]: acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d 2024-04-23T03:21:36.724 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 systemd[1]: Started Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:21:36.724 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:36.687+0000 7f8ea2b93000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:21:37.119 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[26683]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:37.119 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:37.119 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:37.119 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:37.119 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:37.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[21886]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:37.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:37.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:37.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:37.120 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-mon[22716]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:37.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:36.821+0000 7f8ea2b93000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:21:37.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:36.932+0000 7f8ea2b93000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:21:37.602 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T03:21:37.602 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T03:21:37.603 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-23T03:21:37.603 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-23T03:21:37.603 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00849328 s, 1.2 GB/s 2024-04-23T03:21:37.603 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-23T03:21:37.621 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:37.263+0000 7f8ea2b93000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:21:37.621 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:37.350+0000 7f8ea2b93000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:21:38.223 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2024-04-23T03:21:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: Reconfiguring mgr.y (unknown last config time)... 2024-04-23T03:21:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:21:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:21:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: Reconfiguring daemon mgr.y on smithi007 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[21886]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: Reconfiguring mgr.y (unknown last config time)... 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: Reconfiguring daemon mgr.y on smithi007 2024-04-23T03:21:38.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:38 smithi007 ceph-mon[26683]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:38.869 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:38.498+0000 7f8ea2b93000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:21:38.869 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:38.662+0000 7f8ea2b93000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:21:38.869 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:38.747+0000 7f8ea2b93000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:21:38.869 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:38.869 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:38.870 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: Reconfiguring mgr.y (unknown last config time)... 2024-04-23T03:21:38.870 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:21:38.870 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:21:38.870 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:38.870 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: Reconfiguring daemon mgr.y on smithi007 2024-04-23T03:21:38.870 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:38 smithi149 ceph-mon[22716]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:39.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:39.080+0000 7f8ea2b93000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:21:39.522 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:39.174+0000 7f8ea2b93000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:21:39.522 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:39.269+0000 7f8ea2b93000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:21:39.803 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:39.521+0000 7f8ea2b93000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:21:40.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:21:40.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:21:40.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:40.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:39 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:39.966+0000 7f8ea2b93000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:21:40.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:39 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:21:40.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:39 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:21:40.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:39 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:40.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:39 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:21:40.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:39 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:21:40.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:39 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:40.836 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:40.454+0000 7f8ea2b93000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:21:40.836 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:40.546+0000 7f8ea2b93000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:21:40.836 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:40.835+0000 7f8ea2b93000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:21:41.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:40.922+0000 7f8ea2b93000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:21:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:21:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:21:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:21:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:21:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:41.668 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:41 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:41.334+0000 7f8ea2b93000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:21:41.668 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:41 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:41.463+0000 7f8ea2b93000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:21:41.668 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:41 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:41.555+0000 7f8ea2b93000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:21:41.668 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:41 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:41.666+0000 7f8ea2b93000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:21:41.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:41 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:41.754+0000 7f8ea2b93000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:21:42.296 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:42 smithi149 ceph-mon[22716]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:42.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:42 smithi007 ceph-mon[21886]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:42.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:42 smithi007 ceph-mon[26683]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:21:42.665 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:42 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:42.295+0000 7f8ea2b93000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:21:42.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:21:42 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:21:42.661+0000 7f8ea2b93000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:21:43.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[26683]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[21886]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:43.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[21886]: Standby manager daemon x started 2024-04-23T03:21:43.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:21:43.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:21:43.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:21:43.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:43 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:21:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:43 smithi149 ceph-mon[22716]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:43 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:21:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:43 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:21:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:43 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:21:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:43 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:21:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:43 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3703857085' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:21:44.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-23T03:21:44.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:21:44.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2871706183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]: dispatch 2024-04-23T03:21:44.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]: dispatch 2024-04-23T03:21:44.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]': finished 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2409563874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2871706183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]: dispatch 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]: dispatch 2024-04-23T03:21:44.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]': finished 2024-04-23T03:21:44.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T03:21:44.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:21:44.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:44.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:44 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2409563874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2871706183' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]: dispatch 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]: dispatch 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ff109592-82df-4023-97fa-b4ea0339f5fb"}]': finished 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: osdmap e5: 1 total, 0 up, 1 in 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:44.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:44 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2409563874' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:21:47.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:46 smithi007 ceph-mon[21886]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:47.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:46 smithi007 ceph-mon[26683]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:46 smithi149 ceph-mon[22716]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:49.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:48 smithi007 ceph-mon[21886]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:49.196 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:48 smithi007 ceph-mon[26683]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:49.280 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:48 smithi149 ceph-mon[22716]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:50.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:49 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:21:50.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:49 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:50.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:49 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:21:50.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:49 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:50.182 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:49 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:21:50.182 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:49 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:51.180 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:50 smithi007 ceph-mon[21886]: Deploying daemon osd.0 on smithi007 2024-04-23T03:21:51.180 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:50 smithi007 ceph-mon[21886]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:51.180 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:50 smithi007 ceph-mon[26683]: Deploying daemon osd.0 on smithi007 2024-04-23T03:21:51.180 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:50 smithi007 ceph-mon[26683]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:50 smithi149 ceph-mon[22716]: Deploying daemon osd.0 on smithi007 2024-04-23T03:21:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:50 smithi149 ceph-mon[22716]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:52 smithi007 ceph-mon[21886]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:53.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:52 smithi007 ceph-mon[26683]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:53.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:52 smithi149 ceph-mon[22716]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[26683]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:55.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[21886]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:55.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:55.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:55.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:55.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:54 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:54 smithi149 ceph-mon[22716]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:54 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:54 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:54 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:54 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:57.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:56 smithi149 ceph-mon[22716]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:57.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:56 smithi007 ceph-mon[21886]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:57.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:56 smithi007 ceph-mon[26683]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:58.253 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:58.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:58.809 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:58.834 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.0.service 2024-04-23T03:21:58.837 INFO:tasks.cephadm:Deploying osd.1 on smithi007 with /dev/vg_nvme/lv_3... 2024-04-23T03:21:58.837 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T03:21:58.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:21:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:21:59.807 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[21886]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[21886]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[21886]: from='osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[21886]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[26683]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[26683]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[26683]: from='osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:21:59.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-mon[26683]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:21:59.809 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[29530]: 2024-04-23T03:21:59.527+0000 7feca9277700 -1 osd.0 0 waiting for initial osdmap 2024-04-23T03:21:59.809 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:21:59 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[29530]: 2024-04-23T03:21:59.529+0000 7feca440f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:21:59.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:59 smithi149 ceph-mon[22716]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T03:21:59.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:59 smithi149 ceph-mon[22716]: osdmap e6: 1 total, 0 up, 1 in 2024-04-23T03:21:59.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:59 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:21:59.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:59 smithi149 ceph-mon[22716]: from='osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:21:59.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:21:59 smithi149 ceph-mon[22716]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:00.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[26683]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:00.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[26683]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T03:22:00.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:00.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:00.644 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[26683]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:22:00.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[21886]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:00.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[21886]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T03:22:00.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:00.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:00.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:00 smithi007 ceph-mon[21886]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:22:00.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:00 smithi149 ceph-mon[22716]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:00.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:00 smithi149 ceph-mon[22716]: osdmap e7: 1 total, 0 up, 1 in 2024-04-23T03:22:00.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:00 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:00.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:00 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:00.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:00 smithi149 ceph-mon[22716]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-23T03:22:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:22:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:22:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[26683]: osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596] boot 2024-04-23T03:22:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[26683]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T03:22:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:22:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:22:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[21886]: osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596] boot 2024-04-23T03:22:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[21886]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T03:22:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:01 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:01.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:01 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:22:01.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:01 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:22:01.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:01 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:01.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:01 smithi149 ceph-mon[22716]: osd.0 [v2:172.21.15.7:6802/754005596,v1:172.21.15.7:6803/754005596] boot 2024-04-23T03:22:01.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:01 smithi149 ceph-mon[22716]: osdmap e8: 1 total, 1 up, 1 in 2024-04-23T03:22:01.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:01 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:22:02.479 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T03:22:02.480 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T03:22:02.480 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-23T03:22:02.480 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-23T03:22:02.480 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135626 s, 773 MB/s 2024-04-23T03:22:02.480 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-23T03:22:02.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:02 smithi007 ceph-mon[26683]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T03:22:02.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:02 smithi007 ceph-mon[26683]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:02.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:02 smithi007 ceph-mon[21886]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T03:22:02.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:02 smithi007 ceph-mon[21886]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:02 smithi149 ceph-mon[22716]: osdmap e9: 1 total, 1 up, 1 in 2024-04-23T03:22:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:02 smithi149 ceph-mon[22716]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:03.513 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2024-04-23T03:22:04.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi007 2024-04-23T03:22:04.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:04.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi007 to 16069M 2024-04-23T03:22:04.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi007 2024-04-23T03:22:04.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:04.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi007 to 16069M 2024-04-23T03:22:04.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:03 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:03 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi007 2024-04-23T03:22:04.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:03 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:03 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:04.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:03 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi007 to 16069M 2024-04-23T03:22:04.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:03 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:03 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:04.948 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:04 smithi007 ceph-mon[21886]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:04.948 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:04 smithi007 ceph-mon[26683]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:05.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:04 smithi149 ceph-mon[22716]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:06.017 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:05 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:06.017 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:05 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:06.017 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:05 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:06.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:05 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:06.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:05 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:06.018 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:05 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:06.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:05 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:06.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:05 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:06.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:05 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:07.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:06 smithi007 ceph-mon[26683]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:07.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:06 smithi007 ceph-mon[26683]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:07.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:06 smithi007 ceph-mon[21886]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:07.048 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:06 smithi007 ceph-mon[21886]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:07.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:06 smithi149 ceph-mon[22716]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:07.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:06 smithi149 ceph-mon[22716]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:08.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:07 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1854787305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b027e9c2-1825-4603-bc6c-922d3ee8bd3a"}]: dispatch 2024-04-23T03:22:08.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:07 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1854787305' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b027e9c2-1825-4603-bc6c-922d3ee8bd3a"}]': finished 2024-04-23T03:22:08.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:07 smithi149 ceph-mon[22716]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T03:22:08.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:07 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1854787305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b027e9c2-1825-4603-bc6c-922d3ee8bd3a"}]: dispatch 2024-04-23T03:22:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1854787305' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b027e9c2-1825-4603-bc6c-922d3ee8bd3a"}]': finished 2024-04-23T03:22:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[21886]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T03:22:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1854787305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b027e9c2-1825-4603-bc6c-922d3ee8bd3a"}]: dispatch 2024-04-23T03:22:08.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1854787305' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b027e9c2-1825-4603-bc6c-922d3ee8bd3a"}]': finished 2024-04-23T03:22:08.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[26683]: osdmap e10: 2 total, 1 up, 2 in 2024-04-23T03:22:08.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:07 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:09.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:08 smithi149 ceph-mon[22716]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:09.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:08 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1966159857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:09.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:08 smithi007 ceph-mon[21886]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:09.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:08 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1966159857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:09.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:08 smithi007 ceph-mon[26683]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:09.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:08 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1966159857' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:11.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:10 smithi007 ceph-mon[21886]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:11.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:10 smithi007 ceph-mon[26683]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:10 smithi149 ceph-mon[22716]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:13.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:12 smithi007 ceph-mon[21886]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:13.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:12 smithi007 ceph-mon[26683]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:12 smithi149 ceph-mon[22716]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:14.260 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:13 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:22:14.261 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:13 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:14.261 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:13 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:22:14.261 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:13 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:13 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:22:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:13 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:15.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:14 smithi007 ceph-mon[21886]: Deploying daemon osd.1 on smithi007 2024-04-23T03:22:15.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:14 smithi007 ceph-mon[21886]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:15.222 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:14 smithi007 ceph-mon[26683]: Deploying daemon osd.1 on smithi007 2024-04-23T03:22:15.222 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:14 smithi007 ceph-mon[26683]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:14 smithi149 ceph-mon[22716]: Deploying daemon osd.1 on smithi007 2024-04-23T03:22:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:14 smithi149 ceph-mon[22716]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:17.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:16 smithi007 ceph-mon[21886]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:17.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:16 smithi007 ceph-mon[26683]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:17.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:16 smithi149 ceph-mon[22716]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:19.074 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:18 smithi149 ceph-mon[22716]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:19.074 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:18 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:19.075 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:18 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:19.075 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:18 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:19.075 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:18 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[21886]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[26683]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:19.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:18 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:21.143 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:20 smithi007 ceph-mon[21886]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:21.143 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:20 smithi007 ceph-mon[26683]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:21.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:20 smithi149 ceph-mon[22716]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:22.160 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 1 on host 'smithi007' 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:22.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:22.518 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:22.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:22.825 DEBUG:teuthology.orchestra.run.smithi007:osd.1> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.1.service 2024-04-23T03:22:22.828 INFO:tasks.cephadm:Deploying osd.2 on smithi007 with /dev/vg_nvme/lv_2... 2024-04-23T03:22:22.828 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T03:22:23.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[21886]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T03:22:23.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[21886]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T03:22:23.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:23.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[21886]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:23.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[26683]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T03:22:23.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[26683]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T03:22:23.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:23.641 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-mon[26683]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:23.641 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[32501]: 2024-04-23T03:22:23.352+0000 7fa55c83d700 -1 osd.1 0 waiting for initial osdmap 2024-04-23T03:22:23.641 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[32501]: 2024-04-23T03:22:23.355+0000 7fa5551d0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:22:23.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:23 smithi149 ceph-mon[22716]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T03:22:23.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:23 smithi149 ceph-mon[22716]: osdmap e11: 2 total, 1 up, 2 in 2024-04-23T03:22:23.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:23 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:23.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:23 smithi149 ceph-mon[22716]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[21886]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[21886]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T03:22:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[21886]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:24.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[26683]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:24.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[26683]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T03:22:24.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:24.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:24.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:24 smithi007 ceph-mon[26683]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:24 smithi149 ceph-mon[22716]: from='osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:24 smithi149 ceph-mon[22716]: osdmap e12: 2 total, 1 up, 2 in 2024-04-23T03:22:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:24 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:24 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:24 smithi149 ceph-mon[22716]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-23T03:22:25.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:22:25.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:22:25.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[26683]: osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351] boot 2024-04-23T03:22:25.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[26683]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T03:22:25.525 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:25.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:22:25.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:22:25.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[21886]: osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351] boot 2024-04-23T03:22:25.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[21886]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T03:22:25.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:25 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:25 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:22:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:25 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:22:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:25 smithi149 ceph-mon[22716]: osd.1 [v2:172.21.15.7:6810/2742666351,v1:172.21.15.7:6811/2742666351] boot 2024-04-23T03:22:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:25 smithi149 ceph-mon[22716]: osdmap e13: 2 total, 2 up, 2 in 2024-04-23T03:22:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:25 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:22:25.715 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T03:22:25.715 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T03:22:25.716 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-23T03:22:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-23T03:22:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00945097 s, 1.1 GB/s 2024-04-23T03:22:25.716 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-23T03:22:26.404 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2024-04-23T03:22:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:26 smithi149 ceph-mon[22716]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T03:22:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:26 smithi149 ceph-mon[22716]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:26.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:26 smithi007 ceph-mon[21886]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T03:22:26.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:26 smithi007 ceph-mon[21886]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:26.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:26 smithi007 ceph-mon[26683]: osdmap e14: 2 total, 2 up, 2 in 2024-04-23T03:22:26.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:26 smithi007 ceph-mon[26683]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi007 2024-04-23T03:22:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi007 to 8034M 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[21886]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi007 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi007 to 8034M 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.412 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:28 smithi007 ceph-mon[26683]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:28.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi007 2024-04-23T03:22:28.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:28.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:28.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi007 to 8034M 2024-04-23T03:22:28.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:28.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:28 smithi149 ceph-mon[22716]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:29.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[21886]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:29.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:29.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:29.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:29.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[26683]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:29.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:29.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:29.473 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:29 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:29 smithi149 ceph-mon[22716]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:29 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:29 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:29 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:30.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:30 smithi149 ceph-mon[22716]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:30.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:30 smithi007 ceph-mon[21886]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:30.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:30 smithi007 ceph-mon[26683]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:31.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:31 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3529491925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4481e007-05de-4368-9251-336f8c16c038"}]: dispatch 2024-04-23T03:22:31.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:31 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3529491925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4481e007-05de-4368-9251-336f8c16c038"}]': finished 2024-04-23T03:22:31.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:31 smithi149 ceph-mon[22716]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T03:22:31.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:31 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3529491925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4481e007-05de-4368-9251-336f8c16c038"}]: dispatch 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3529491925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4481e007-05de-4368-9251-336f8c16c038"}]': finished 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[21886]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3529491925' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4481e007-05de-4368-9251-336f8c16c038"}]: dispatch 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3529491925' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4481e007-05de-4368-9251-336f8c16c038"}]': finished 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[26683]: osdmap e15: 3 total, 2 up, 3 in 2024-04-23T03:22:31.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:31 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:32.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:32 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/468891898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:32.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:32 smithi149 ceph-mon[22716]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:32 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/468891898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:32 smithi007 ceph-mon[26683]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:32.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:32 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/468891898' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:32.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:32 smithi007 ceph-mon[21886]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:34 smithi149 ceph-mon[22716]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:35.187 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:34 smithi007 ceph-mon[26683]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:35.187 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:34 smithi007 ceph-mon[21886]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:37.272 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:36 smithi007 ceph-mon[21886]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:37.272 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:36 smithi007 ceph-mon[26683]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:36 smithi149 ceph-mon[22716]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:38.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:37 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:22:38.033 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:37 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:38.033 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:37 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:22:38.033 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:37 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:38.323 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:37 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:22:38.323 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:37 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:39.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:38 smithi007 ceph-mon[21886]: Deploying daemon osd.2 on smithi007 2024-04-23T03:22:39.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:38 smithi007 ceph-mon[21886]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:38 smithi007 ceph-mon[26683]: Deploying daemon osd.2 on smithi007 2024-04-23T03:22:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:38 smithi007 ceph-mon[26683]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:38 smithi149 ceph-mon[22716]: Deploying daemon osd.2 on smithi007 2024-04-23T03:22:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:38 smithi149 ceph-mon[22716]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:41.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:40 smithi007 ceph-mon[21886]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:41.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:22:41.308 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:22:41.308 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:40 smithi007 ceph-mon[26683]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:41.308 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:22:41.308 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:22:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:40 smithi149 ceph-mon[22716]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:22:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:22:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[21886]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[26683]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:43.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:42 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:42 smithi149 ceph-mon[22716]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:42 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:42 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:42 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:42 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:45.323 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:44 smithi007 ceph-mon[21886]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:45.323 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:45.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:45.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:44 smithi007 ceph-mon[26683]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:45.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:45.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:44 smithi149 ceph-mon[22716]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:45.659 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 2 on host 'smithi007' 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[21886]: from='osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[21886]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[26683]: from='osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[26683]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:46.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:46.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:46.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:45 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:46.350 DEBUG:teuthology.orchestra.run.smithi007:osd.2> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.2.service 2024-04-23T03:22:46.352 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_1... 2024-04-23T03:22:46.352 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T03:22:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:45 smithi149 ceph-mon[22716]: from='osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:22:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:45 smithi149 ceph-mon[22716]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:22:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:45 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:45 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:22:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:45 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:45 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:22:47.203 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[26683]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[26683]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[26683]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[26683]: from='osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[26683]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[21886]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[21886]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[21886]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T03:22:47.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:47.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[21886]: from='osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:47.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-mon[21886]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:47.205 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[35443]: 2024-04-23T03:22:46.946+0000 7f91a2ad4700 -1 osd.2 0 waiting for initial osdmap 2024-04-23T03:22:47.205 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[35443]: 2024-04-23T03:22:46.949+0000 7f9199c64700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:22:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:46 smithi149 ceph-mon[22716]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:46 smithi149 ceph-mon[22716]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T03:22:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:46 smithi149 ceph-mon[22716]: osdmap e16: 3 total, 2 up, 3 in 2024-04-23T03:22:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:46 smithi149 ceph-mon[22716]: from='osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:47.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:46 smithi149 ceph-mon[22716]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:22:48.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[26683]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:48.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[26683]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T03:22:48.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:48.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:48.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[21886]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:48.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[21886]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T03:22:48.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:48.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:47 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:48.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:47 smithi149 ceph-mon[22716]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:22:48.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:47 smithi149 ceph-mon[22716]: osdmap e17: 3 total, 2 up, 3 in 2024-04-23T03:22:48.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:47 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:48.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:47 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:22:49.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:22:49.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:49.266 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933] boot 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933] boot 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:48 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-23T03:22:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: osd.2 [v2:172.21.15.7:6818/1063967933,v1:172.21.15.7:6819/1063967933] boot 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: osdmap e18: 3 total, 3 up, 3 in 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:22:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:48 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-23T03:22:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T03:22:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T03:22:49.804 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-23T03:22:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-23T03:22:49.804 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00961441 s, 1.1 GB/s 2024-04-23T03:22:49.804 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[21886]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[21886]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[26683]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T03:22:50.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T03:22:50.062 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:49 smithi007 ceph-mon[26683]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T03:22:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:49 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-23T03:22:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:49 smithi149 ceph-mon[22716]: osdmap e19: 3 total, 3 up, 3 in 2024-04-23T03:22:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:49 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-23T03:22:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:49 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-23T03:22:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:49 smithi149 ceph-mon[22716]: osdmap e20: 3 total, 3 up, 3 in 2024-04-23T03:22:50.725 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:22:50 smithi007 sudo[36669]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T03:22:50.725 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:22:50 smithi007 sudo[36669]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:50.725 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:22:50 smithi007 sudo[36669]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:51.076 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:22:50 smithi007 sudo[36672]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T03:22:51.076 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:22:50 smithi007 sudo[36672]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:51.076 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:22:50 smithi007 sudo[36672]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:51.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:51.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.076 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[26683]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:50 smithi007 ceph-mon[21886]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T03:22:51.099 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2024-04-23T03:22:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: pgmap v74: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:22:51.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:51.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:50 smithi149 ceph-mon[22716]: osdmap e21: 3 total, 3 up, 3 in 2024-04-23T03:22:51.695 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:51 smithi007 sudo[36720]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T03:22:51.695 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:51 smithi007 sudo[36720]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:51.695 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:51 smithi007 sudo[36720]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:51.961 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:51 smithi007 sudo[36740]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T03:22:51.961 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:51 smithi007 sudo[36740]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:51.961 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:22:51 smithi007 sudo[36740]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:52.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:51 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi007 2024-04-23T03:22:52.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:51 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi007 to 5356M 2024-04-23T03:22:52.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:51 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:52.302 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:51 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi007 2024-04-23T03:22:52.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:51 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi007 to 5356M 2024-04-23T03:22:52.303 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:51 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:51 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi007 2024-04-23T03:22:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:51 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi007 to 5356M 2024-04-23T03:22:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:51 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:52.557 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:52 smithi007 sudo[36746]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T03:22:52.557 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:52 smithi007 sudo[36746]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:52.558 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:52 smithi007 sudo[36746]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:52.833 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:52 smithi007 sudo[36774]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T03:22:52.833 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:52 smithi007 sudo[36774]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:52.833 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:22:52 smithi007 sudo[36774]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 sudo[36800]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T03:22:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 sudo[36800]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 sudo[36800]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:53 smithi149 ceph-mon[22716]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:53 smithi149 ceph-mon[22716]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-23T03:22:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:53 smithi149 ceph-mon[22716]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:54.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:53 smithi007 ceph-mon[26683]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:54.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:53 smithi007 ceph-mon[26683]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-23T03:22:54.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:53 smithi007 ceph-mon[26683]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:54.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 sudo[36803]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T03:22:54.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 sudo[36803]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:54.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 sudo[36803]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:54.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 ceph-mon[21886]: pgmap v77: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:54.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 ceph-mon[21886]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-23T03:22:54.038 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:53 smithi007 ceph-mon[21886]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:54.288 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:54 smithi007 sudo[36882]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T03:22:54.288 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:54 smithi007 sudo[36882]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:54.288 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:54 smithi007 sudo[36882]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:54 smithi007 sudo[36906]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T03:22:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:54 smithi007 sudo[36906]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:54 smithi007 sudo[36906]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:54 smithi149 sudo[24234]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:54 smithi149 sudo[24234]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:54 smithi149 sudo[24234]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 sudo[24237]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 sudo[24237]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 sudo[24237]: pam_unix(sudo:session): session closed for user root 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:55 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:55.182 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='client.24211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.183 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: pgmap v78: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:22:55.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:55 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:22:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 263 KiB/s wr, 18 op/s 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/93107156' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a4c8dda-e151-41cd-9e1c-e96b64711326"}]: dispatch 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/93107156' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a4c8dda-e151-41cd-9e1c-e96b64711326"}]': finished 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:56.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 263 KiB/s wr, 18 op/s 2024-04-23T03:22:56.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/93107156' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a4c8dda-e151-41cd-9e1c-e96b64711326"}]: dispatch 2024-04-23T03:22:56.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/93107156' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a4c8dda-e151-41cd-9e1c-e96b64711326"}]': finished 2024-04-23T03:22:56.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T03:22:56.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:22:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 263 KiB/s wr, 18 op/s 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/93107156' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a4c8dda-e151-41cd-9e1c-e96b64711326"}]: dispatch 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/93107156' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a4c8dda-e151-41cd-9e1c-e96b64711326"}]': finished 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: osdmap e22: 4 total, 3 up, 4 in 2024-04-23T03:22:56.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:22:57.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:57 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1705644654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:57.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:57 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1705644654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:57 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1705644654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:22:58.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:22:58 smithi149 ceph-mon[22716]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 187 KiB/s wr, 12 op/s 2024-04-23T03:22:58.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:22:58 smithi007 ceph-mon[21886]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 187 KiB/s wr, 12 op/s 2024-04-23T03:22:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:22:58 smithi007 ceph-mon[26683]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 187 KiB/s wr, 12 op/s 2024-04-23T03:23:01.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:00 smithi007 ceph-mon[26683]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 146 KiB/s wr, 10 op/s 2024-04-23T03:23:01.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:00 smithi007 ceph-mon[21886]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 146 KiB/s wr, 10 op/s 2024-04-23T03:23:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:00 smithi149 ceph-mon[22716]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 146 KiB/s wr, 10 op/s 2024-04-23T03:23:03.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:02 smithi007 ceph-mon[26683]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-23T03:23:03.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:02 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:23:03.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:02 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:03.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:02 smithi007 ceph-mon[21886]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-23T03:23:03.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:02 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:23:03.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:02 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:02 smithi149 ceph-mon[22716]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-23T03:23:03.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:02 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:23:03.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:02 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:04.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:03 smithi007 ceph-mon[21886]: Deploying daemon osd.3 on smithi007 2024-04-23T03:23:04.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:03 smithi007 ceph-mon[26683]: Deploying daemon osd.3 on smithi007 2024-04-23T03:23:04.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:03 smithi149 ceph-mon[22716]: Deploying daemon osd.3 on smithi007 2024-04-23T03:23:05.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:04 smithi007 ceph-mon[26683]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-23T03:23:05.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:04 smithi007 ceph-mon[21886]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-23T03:23:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:04 smithi149 ceph-mon[22716]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-23T03:23:07.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:06 smithi007 ceph-mon[26683]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:07.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:06 smithi007 ceph-mon[21886]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:06 smithi149 ceph-mon[22716]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:08.324 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:08 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:08.324 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:08 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:08.324 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:08 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:08.324 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:08 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:08.324 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:08 smithi149 ceph-mon[22716]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:08.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:08.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:08.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:08.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[21886]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:08.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:08.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:08.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:08.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:08.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:08 smithi007 ceph-mon[26683]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:10.288 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2024-04-23T03:23:10.892 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.3.service 2024-04-23T03:23:10.894 INFO:tasks.cephadm:Deploying osd.4 on smithi149 with /dev/vg_nvme/lv_4... 2024-04-23T03:23:10.895 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-23T03:23:11.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:10 smithi149 ceph-mon[22716]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:11.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:10 smithi149 ceph-mon[22716]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:23:11.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:10 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:11.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:10 smithi007 ceph-mon[26683]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:11.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:10 smithi007 ceph-mon[26683]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:23:11.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:10 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:11.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:10 smithi007 ceph-mon[21886]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:11.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:10 smithi007 ceph-mon[21886]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:23:11.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:10 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:12.120 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:11 smithi149 ceph-mon[22716]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T03:23:12.120 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:11 smithi149 ceph-mon[22716]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T03:23:12.120 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:11 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:12.120 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:11 smithi149 ceph-mon[22716]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:23:12.120 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:11 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:12.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[26683]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T03:23:12.324 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[26683]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T03:23:12.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:12.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[26683]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:23:12.325 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:12.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[21886]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T03:23:12.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[21886]: osdmap e23: 4 total, 3 up, 4 in 2024-04-23T03:23:12.325 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:12.326 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[21886]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:23:12.326 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:12.326 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[38426]: 2024-04-23T03:23:11.933+0000 7f4c18d7c700 -1 osd.3 0 waiting for initial osdmap 2024-04-23T03:23:12.326 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:23:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[38426]: 2024-04-23T03:23:11.938+0000 7f4c14f16700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:23:12.897 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-23T03:23:12.897 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-23T03:23:12.898 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-04-23T03:23:12.898 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-04-23T03:23:12.898 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128199 s, 818 MB/s 2024-04-23T03:23:12.898 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-04-23T03:23:13.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:12 smithi149 ceph-mon[22716]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:13.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:12 smithi149 ceph-mon[22716]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:23:13.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:12 smithi149 ceph-mon[22716]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T03:23:13.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:12 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:13.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:12 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[21886]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[21886]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:23:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[21886]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[26683]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[26683]: from='osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[26683]: osdmap e24: 4 total, 3 up, 4 in 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:12 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:13.499 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi149:vg_nvme/lv_4 2024-04-23T03:23:13.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:23:14.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:13 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:23:14.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:13 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:23:14.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:13 smithi149 ceph-mon[22716]: osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379] boot 2024-04-23T03:23:14.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:13 smithi149 ceph-mon[22716]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T03:23:14.054 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:13 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:14.317 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[26683]: osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379] boot 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[26683]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[21886]: osd.3 [v2:172.21.15.7:6826/3422810379,v1:172.21.15.7:6827/3422810379] boot 2024-04-23T03:23:14.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[21886]: osdmap e25: 4 total, 4 up, 4 in 2024-04-23T03:23:14.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:13 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:23:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T03:23:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:15.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:15.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:15.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:14 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: osdmap e26: 4 total, 4 up, 4 in 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.345 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:15.346 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:15.346 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:15.346 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:15.346 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:14 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:16.332 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:15 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi007 2024-04-23T03:23:16.332 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:15 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi007 to 4017M 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:15 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:15 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:15 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi007 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi007 to 4017M 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:23:16.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:16.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi007 2024-04-23T03:23:16.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi007 to 4017M 2024-04-23T03:23:16.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:23:16.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:23:16.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:15 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:17.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:16 smithi149 ceph-mon[22716]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:23:17.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:16 smithi149 ceph-mon[22716]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:17.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:16 smithi007 ceph-mon[26683]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:23:17.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:16 smithi007 ceph-mon[26683]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:17.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:16 smithi007 ceph-mon[21886]: from='client.34190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:23:17.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:16 smithi007 ceph-mon[21886]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:18.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/2106471357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]: dispatch 2024-04-23T03:23:18.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]: dispatch 2024-04-23T03:23:18.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]': finished 2024-04-23T03:23:18.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[21886]: osdmap e27: 5 total, 4 up, 5 in 2024-04-23T03:23:18.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:18.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/2106471357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]: dispatch 2024-04-23T03:23:18.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]: dispatch 2024-04-23T03:23:18.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]': finished 2024-04-23T03:23:18.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[26683]: osdmap e27: 5 total, 4 up, 5 in 2024-04-23T03:23:18.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:17 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:17 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/2106471357' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]: dispatch 2024-04-23T03:23:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:17 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]: dispatch 2024-04-23T03:23:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:17 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ceeb2e6-0fcf-48e1-995c-10fd315354c9"}]': finished 2024-04-23T03:23:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:17 smithi149 ceph-mon[22716]: osdmap e27: 5 total, 4 up, 5 in 2024-04-23T03:23:18.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:17 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:18 smithi007 ceph-mon[21886]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:18 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/2321381390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:23:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:18 smithi007 ceph-mon[26683]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:18 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/2321381390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:23:19.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:18 smithi149 ceph-mon[22716]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:19.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:18 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/2321381390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:23:21.289 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:20 smithi149 ceph-mon[22716]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:20 smithi007 ceph-mon[26683]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:20 smithi007 ceph-mon[21886]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:23.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:22 smithi149 ceph-mon[22716]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:22 smithi007 ceph-mon[21886]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:22 smithi007 ceph-mon[26683]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:24.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:23 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:23:24.157 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:23 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:23 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:23:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:23 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:23 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:23:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:23 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:24.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:24 smithi149 ceph-mon[22716]: Deploying daemon osd.4 on smithi149 2024-04-23T03:23:24.986 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:24 smithi149 ceph-mon[22716]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:24 smithi007 ceph-mon[26683]: Deploying daemon osd.4 on smithi149 2024-04-23T03:23:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:24 smithi007 ceph-mon[26683]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:24 smithi007 ceph-mon[21886]: Deploying daemon osd.4 on smithi149 2024-04-23T03:23:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:24 smithi007 ceph-mon[21886]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:27.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:26 smithi149 ceph-mon[22716]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:27.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:26 smithi007 ceph-mon[21886]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:27.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:26 smithi007 ceph-mon[26683]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:29.132 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:28 smithi149 ceph-mon[22716]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:29.132 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:29.132 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:29.132 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:29.132 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:28 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[21886]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[26683]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:29.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:29.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:29.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:28 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:31.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:30 smithi007 ceph-mon[21886]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:30 smithi007 ceph-mon[26683]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:31.387 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:30 smithi149 ceph-mon[22716]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:31.416 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 4 on host 'smithi149' 2024-04-23T03:23:31.946 DEBUG:teuthology.orchestra.run.smithi149:osd.4> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.4.service 2024-04-23T03:23:31.949 INFO:tasks.cephadm:Deploying osd.5 on smithi149 with /dev/vg_nvme/lv_3... 2024-04-23T03:23:31.950 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-23T03:23:32.559 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:32 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:32.560 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:32 smithi149 ceph-mon[22716]: from='osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:23:32.560 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:32 smithi149 ceph-mon[22716]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:23:32.560 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:32 smithi149 ceph-mon[22716]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:32.560 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:32 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:32.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:32.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[21886]: from='osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:23:32.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[21886]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:23:32.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[21886]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:32.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:32.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[26683]: from='osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:23:32.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[26683]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:23:32.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[26683]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:32.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:32 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:33.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-mon[22716]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T03:23:33.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-mon[22716]: osdmap e28: 5 total, 4 up, 5 in 2024-04-23T03:23:33.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:33.489 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-mon[22716]: from='osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:33.490 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-mon[22716]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:33.490 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[26091]: 2024-04-23T03:23:33.427+0000 7f03d011f700 -1 osd.4 0 waiting for initial osdmap 2024-04-23T03:23:33.490 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:23:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[26091]: 2024-04-23T03:23:33.431+0000 7f03c92b3700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:23:33.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[26683]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T03:23:33.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[26683]: osdmap e28: 5 total, 4 up, 5 in 2024-04-23T03:23:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[26683]: from='osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:33.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[26683]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[21886]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T03:23:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[21886]: osdmap e28: 5 total, 4 up, 5 in 2024-04-23T03:23:33.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:33.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[21886]: from='osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:33.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:33 smithi007 ceph-mon[21886]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:33.954 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-23T03:23:33.954 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-23T03:23:33.954 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-04-23T03:23:33.954 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-04-23T03:23:33.955 INFO:teuthology.orchestra.run.smithi149.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130682 s, 802 MB/s 2024-04-23T03:23:33.955 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: osdmap e29: 5 total, 4 up, 5 in 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787] boot 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: osdmap e30: 5 total, 5 up, 5 in 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.735 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: osdmap e29: 5 total, 4 up, 5 in 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787] boot 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: osdmap e30: 5 total, 5 up, 5 in 2024-04-23T03:23:34.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:34 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.866 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:23:34.866 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: osdmap e29: 5 total, 4 up, 5 in 2024-04-23T03:23:34.866 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.866 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:34.867 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-23T03:23:34.867 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: osd.4 [v2:172.21.15.149:6800/1331461787,v1:172.21.15.149:6801/1331461787] boot 2024-04-23T03:23:34.867 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: osdmap e30: 5 total, 5 up, 5 in 2024-04-23T03:23:34.867 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:34 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:23:35.731 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:35 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:23:35.731 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:35 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:23:35.731 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:35 smithi149 ceph-mon[22716]: osdmap e31: 5 total, 5 up, 5 in 2024-04-23T03:23:35.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:35 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:23:35.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:35 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:23:35.751 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:35 smithi007 ceph-mon[21886]: osdmap e31: 5 total, 5 up, 5 in 2024-04-23T03:23:35.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:35 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:23:35.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:35 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:23:35.752 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:35 smithi007 ceph-mon[26683]: osdmap e31: 5 total, 5 up, 5 in 2024-04-23T03:23:35.754 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi149:vg_nvme/lv_3 2024-04-23T03:23:37.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:37.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: osdmap e32: 5 total, 5 up, 5 in 2024-04-23T03:23:37.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:37.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:37.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:37.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:37.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:36 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: osdmap e32: 5 total, 5 up, 5 in 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: osdmap e32: 5 total, 5 up, 5 in 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:37.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:36 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:37.970 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:37 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi149 2024-04-23T03:23:37.971 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:37 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 5816M 2024-04-23T03:23:37.971 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:37 smithi149 ceph-mon[22716]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T03:23:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:37 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi149 2024-04-23T03:23:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:37 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi149 to 5816M 2024-04-23T03:23:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:37 smithi007 ceph-mon[21886]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T03:23:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:37 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi149 2024-04-23T03:23:38.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:37 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 5816M 2024-04-23T03:23:38.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:37 smithi007 ceph-mon[26683]: osdmap e33: 5 total, 5 up, 5 in 2024-04-23T03:23:39.014 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:38 smithi149 ceph-mon[22716]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:39.014 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:38 smithi149 ceph-mon[22716]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T03:23:39.014 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:23:39.014 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:23:39.014 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[21886]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[21886]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T03:23:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:23:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:23:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[26683]: pgmap v112: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[26683]: osdmap e34: 5 total, 5 up, 5 in 2024-04-23T03:23:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:23:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:23:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:40.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:39 smithi149 ceph-mon[22716]: from='client.34217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:23:40.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:39 smithi007 ceph-mon[21886]: from='client.34217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:23:40.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:39 smithi007 ceph-mon[26683]: from='client.34217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/2540737671' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]: dispatch 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]: dispatch 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]': finished 2024-04-23T03:23:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/2540737671' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]: dispatch 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]: dispatch 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]': finished 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T03:23:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:40 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/2540737671' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]: dispatch 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]: dispatch 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed2557da-b879-4a5d-bcf7-0c291bb47b74"}]': finished 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: osdmap e35: 6 total, 5 up, 6 in 2024-04-23T03:23:41.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:40 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:42.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:41 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/1768960854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:23:42.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:41 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/1768960854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:23:42.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:41 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/1768960854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:23:43.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:42 smithi007 ceph-mon[26683]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:23:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:42 smithi007 ceph-mon[21886]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:23:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:42 smithi149 ceph-mon[22716]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:23:45.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:44 smithi149 ceph-mon[22716]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-23T03:23:45.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:44 smithi007 ceph-mon[21886]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-23T03:23:45.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:44 smithi007 ceph-mon[26683]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-23T03:23:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:46 smithi149 ceph-mon[22716]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T03:23:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:23:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:46 smithi007 ceph-mon[21886]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T03:23:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:23:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:47.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:46 smithi007 ceph-mon[26683]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T03:23:47.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:23:47.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:48.266 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:47 smithi149 ceph-mon[22716]: Deploying daemon osd.5 on smithi149 2024-04-23T03:23:48.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:47 smithi007 ceph-mon[21886]: Deploying daemon osd.5 on smithi149 2024-04-23T03:23:48.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:47 smithi007 ceph-mon[26683]: Deploying daemon osd.5 on smithi149 2024-04-23T03:23:49.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:48 smithi149 ceph-mon[22716]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:23:49.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:48 smithi007 ceph-mon[21886]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:23:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:48 smithi007 ceph-mon[26683]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:23:51.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:50 smithi007 ceph-mon[26683]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:23:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:50 smithi007 ceph-mon[21886]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:23:51.393 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:50 smithi149 ceph-mon[22716]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:23:53.045 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:53.045 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:53.045 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:53.045 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:53.046 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:52 smithi149 ceph-mon[22716]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[26683]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:52 smithi007 ceph-mon[21886]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:55.099 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:54 smithi149 ceph-mon[22716]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:55.099 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:54 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:55.099 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:54 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:55.099 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:54 smithi149 ceph-mon[22716]: from='osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:23:55.099 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:54 smithi149 ceph-mon[22716]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:23:55.108 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 5 on host 'smithi149' 2024-04-23T03:23:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[26683]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[26683]: from='osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:23:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[26683]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:23:55.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[21886]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:55.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:55.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:55.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[21886]: from='osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:23:55.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:54 smithi007 ceph-mon[21886]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:23:55.751 DEBUG:teuthology.orchestra.run.smithi149:osd.5> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.5.service 2024-04-23T03:23:55.754 INFO:tasks.cephadm:Deploying osd.6 on smithi149 with /dev/vg_nvme/lv_2... 2024-04-23T03:23:55.754 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:56.244 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:56.245 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-mon[22716]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:56.562 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[29040]: 2024-04-23T03:23:56.241+0000 7f0dc7139700 -1 osd.5 0 waiting for initial osdmap 2024-04-23T03:23:56.562 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:23:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[29040]: 2024-04-23T03:23:56.246+0000 7f0dc2ad2700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:23:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[26683]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:23:56.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T03:23:56.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: osdmap e36: 6 total, 5 up, 6 in 2024-04-23T03:23:56.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:56.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:56.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:23:56.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:56 smithi007 ceph-mon[21886]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-23T03:23:57.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:57 smithi149 ceph-mon[22716]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:23:57.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:57 smithi149 ceph-mon[22716]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T03:23:57.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:57 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:57.495 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:57 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[26683]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[26683]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[21886]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[21886]: osdmap e37: 6 total, 5 up, 6 in 2024-04-23T03:23:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:57.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:57 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:58.340 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:58 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:23:58.340 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:58 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:23:58.340 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:58 smithi149 ceph-mon[22716]: osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801] boot 2024-04-23T03:23:58.340 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:58 smithi149 ceph-mon[22716]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T03:23:58.340 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:58 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:58.340 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:58 smithi149 ceph-mon[22716]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:23:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:23:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:23:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[26683]: osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801] boot 2024-04-23T03:23:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[26683]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[26683]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[21886]: osd.5 [v2:172.21.15.149:6808/2210178801,v1:172.21.15.149:6809/2210178801] boot 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[21886]: osdmap e38: 6 total, 6 up, 6 in 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:23:58.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:58 smithi007 ceph-mon[21886]: pgmap v127: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T03:23:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T03:23:59.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:23:59 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: osdmap e39: 6 total, 6 up, 6 in 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: osdmap e40: 6 total, 6 up, 6 in 2024-04-23T03:23:59.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:23:59 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:23:59.707 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-23T03:23:59.708 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-23T03:23:59.708 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-04-23T03:23:59.708 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-04-23T03:23:59.708 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113994 s, 920 MB/s 2024-04-23T03:23:59.708 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-04-23T03:24:00.329 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:00 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi149 2024-04-23T03:24:00.329 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:00 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 2908M 2024-04-23T03:24:00.329 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:00 smithi149 ceph-mon[22716]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:00.352 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi149:vg_nvme/lv_2 2024-04-23T03:24:00.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:00 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi149 2024-04-23T03:24:00.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:00 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi149 to 2908M 2024-04-23T03:24:00.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:00 smithi007 ceph-mon[21886]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:00.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:00 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi149 2024-04-23T03:24:00.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:00 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 2908M 2024-04-23T03:24:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:00 smithi007 ceph-mon[26683]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:01.538 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:01 smithi149 ceph-mon[22716]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T03:24:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:01 smithi007 ceph-mon[26683]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T03:24:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:01 smithi007 ceph-mon[21886]: osdmap e41: 6 total, 6 up, 6 in 2024-04-23T03:24:02.536 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:02 smithi149 ceph-mon[22716]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T03:24:02.536 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:02 smithi149 ceph-mon[22716]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:02.536 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:02 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:24:02.536 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:02 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:24:02.536 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:02 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[26683]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[26683]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[21886]: osdmap e42: 6 total, 6 up, 6 in 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[21886]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:24:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:02 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:03 smithi149 ceph-mon[22716]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:03.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:03 smithi007 ceph-mon[26683]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:03.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:03 smithi007 ceph-mon[21886]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: Cluster is now healthy 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/2439176630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]: dispatch 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]: dispatch 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]': finished 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T03:24:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: Cluster is now healthy 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/2439176630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]: dispatch 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]: dispatch 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]': finished 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T03:24:05.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:04 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-23T03:24:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-23T03:24:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: Cluster is now healthy 2024-04-23T03:24:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/2439176630' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]: dispatch 2024-04-23T03:24:05.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]: dispatch 2024-04-23T03:24:05.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0c5b5546-4eac-4366-8b18-298c2ece3795"}]': finished 2024-04-23T03:24:05.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: osdmap e43: 7 total, 6 up, 7 in 2024-04-23T03:24:05.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:04 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:06.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:05 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/1132460499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:24:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:05 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/1132460499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:24:06.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:05 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/1132460499' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:24:07.253 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:06 smithi149 ceph-mon[22716]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:24:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:06 smithi007 ceph-mon[21886]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:24:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:06 smithi007 ceph-mon[26683]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:24:09.121 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:08 smithi149 ceph-mon[22716]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T03:24:09.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:08 smithi007 ceph-mon[21886]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T03:24:09.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:08 smithi007 ceph-mon[26683]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-04-23T03:24:11.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:10 smithi149 ceph-mon[22716]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-23T03:24:11.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:10 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:24:11.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:10 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:10 smithi007 ceph-mon[21886]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-23T03:24:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:10 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:24:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:10 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:11.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:10 smithi007 ceph-mon[26683]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-23T03:24:11.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:10 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:24:11.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:10 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:12.274 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:11 smithi149 ceph-mon[22716]: Deploying daemon osd.6 on smithi149 2024-04-23T03:24:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:11 smithi007 ceph-mon[21886]: Deploying daemon osd.6 on smithi149 2024-04-23T03:24:12.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:11 smithi007 ceph-mon[26683]: Deploying daemon osd.6 on smithi149 2024-04-23T03:24:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:12 smithi149 ceph-mon[22716]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:24:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:12 smithi007 ceph-mon[21886]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:24:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:12 smithi007 ceph-mon[26683]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:24:15.223 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:14 smithi149 ceph-mon[22716]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:15.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:14 smithi007 ceph-mon[26683]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:15.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:14 smithi007 ceph-mon[21886]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:16.542 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:16 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:16.543 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:16 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:16.543 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:16 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:16.543 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:16 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:16.543 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:16 smithi149 ceph-mon[22716]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[26683]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:16.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:16.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:16.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:16.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:16 smithi007 ceph-mon[21886]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:17.867 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 6 on host 'smithi149' 2024-04-23T03:24:18.901 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:18 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:18.901 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:18 smithi149 ceph-mon[22716]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:18.901 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:18 smithi149 ceph-mon[22716]: from='osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:24:18.901 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:18 smithi149 ceph-mon[22716]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:24:18.902 DEBUG:teuthology.orchestra.run.smithi149:osd.6> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.6.service 2024-04-23T03:24:18.904 INFO:tasks.cephadm:Deploying osd.7 on smithi149 with /dev/vg_nvme/lv_1... 2024-04-23T03:24:18.905 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-23T03:24:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[21886]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[21886]: from='osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:24:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[21886]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:24:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:19.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[26683]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:19.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[26683]: from='osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:24:19.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:18 smithi007 ceph-mon[26683]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:24:20.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-mon[22716]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T03:24:20.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-mon[22716]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T03:24:20.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:20.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-mon[22716]: from='osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:20.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-mon[22716]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:20.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:20.148 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[31947]: 2024-04-23T03:24:19.877+0000 7f0d1a6c3700 -1 osd.6 0 waiting for initial osdmap 2024-04-23T03:24:20.148 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:24:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[31947]: 2024-04-23T03:24:19.885+0000 7f0d1505a700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[21886]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[21886]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[21886]: from='osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[21886]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[26683]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[26683]: osdmap e44: 7 total, 6 up, 7 in 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:20.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[26683]: from='osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:20.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[26683]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:20.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:19 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:20 smithi149 ceph-mon[22716]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:24:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:20 smithi149 ceph-mon[22716]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T03:24:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:20 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:20.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:20 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:20.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:20 smithi149 ceph-mon[22716]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[26683]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:24:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[26683]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T03:24:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[26683]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[21886]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:24:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[21886]: osdmap e45: 7 total, 6 up, 7 in 2024-04-23T03:24:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:20 smithi007 ceph-mon[21886]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-23T03:24:22.129 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:21 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:24:22.129 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:21 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:24:22.129 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:21 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:22.129 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:21 smithi149 ceph-mon[22716]: osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633] boot 2024-04-23T03:24:22.129 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:21 smithi149 ceph-mon[22716]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T03:24:22.129 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:21 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:22.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:24:22.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:24:22.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:22.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[21886]: osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633] boot 2024-04-23T03:24:22.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[21886]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T03:24:22.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:22.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:24:22.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:24:22.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:22.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[26683]: osd.6 [v2:172.21.15.149:6816/3021260633,v1:172.21.15.149:6817/3021260633] boot 2024-04-23T03:24:22.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[26683]: osdmap e46: 7 total, 7 up, 7 in 2024-04-23T03:24:22.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:21 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:24:22.513 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-23T03:24:22.514 INFO:teuthology.orchestra.run.smithi149.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-23T03:24:22.514 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10+0 records in 2024-04-23T03:24:22.514 INFO:teuthology.orchestra.run.smithi149.stderr:10+0 records out 2024-04-23T03:24:22.514 INFO:teuthology.orchestra.run.smithi149.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126056 s, 832 MB/s 2024-04-23T03:24:22.514 INFO:teuthology.orchestra.run.smithi149.stderr:--> Zapping successful for: 2024-04-23T03:24:23.167 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:23.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:22 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.191 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi149:vg_nvme/lv_1 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: osdmap e47: 7 total, 7 up, 7 in 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: pgmap v148: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:23.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:23.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:23.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:22 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:24.165 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:23 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi149 2024-04-23T03:24:24.165 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:23 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 1938M 2024-04-23T03:24:24.165 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:23 smithi149 ceph-mon[22716]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T03:24:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:23 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi149 2024-04-23T03:24:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:23 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 1938M 2024-04-23T03:24:24.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:23 smithi007 ceph-mon[26683]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T03:24:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:23 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi149 2024-04-23T03:24:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:23 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi149 to 1938M 2024-04-23T03:24:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:23 smithi007 ceph-mon[21886]: osdmap e48: 7 total, 7 up, 7 in 2024-04-23T03:24:25.081 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:24 smithi149 ceph-mon[22716]: osdmap e49: 7 total, 7 up, 7 in 2024-04-23T03:24:25.081 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:24 smithi149 ceph-mon[22716]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:24 smithi007 ceph-mon[21886]: osdmap e49: 7 total, 7 up, 7 in 2024-04-23T03:24:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:24 smithi007 ceph-mon[21886]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:24 smithi007 ceph-mon[26683]: osdmap e49: 7 total, 7 up, 7 in 2024-04-23T03:24:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:24 smithi007 ceph-mon[26683]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:26.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:25 smithi149 ceph-mon[22716]: osdmap e50: 7 total, 7 up, 7 in 2024-04-23T03:24:26.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:25 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:24:26.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:25 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:24:26.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:25 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:26.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[21886]: osdmap e50: 7 total, 7 up, 7 in 2024-04-23T03:24:26.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:24:26.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:24:26.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:26.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[26683]: osdmap e50: 7 total, 7 up, 7 in 2024-04-23T03:24:26.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-23T03:24:26.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-23T03:24:26.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:25 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:27.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:26 smithi149 ceph-mon[22716]: from='client.34271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:27.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:26 smithi149 ceph-mon[22716]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:27.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:26 smithi007 ceph-mon[26683]: from='client.34271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:27.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:26 smithi007 ceph-mon[26683]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:27.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:26 smithi007 ceph-mon[21886]: from='client.34271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi149:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:27.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:26 smithi007 ceph-mon[21886]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:28.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:27 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/3928369250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]: dispatch 2024-04-23T03:24:28.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:27 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]: dispatch 2024-04-23T03:24:28.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:27 smithi149 ceph-mon[22716]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]': finished 2024-04-23T03:24:28.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:27 smithi149 ceph-mon[22716]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T03:24:28.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:27 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:28.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:27 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/44222832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:24:28.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/3928369250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]: dispatch 2024-04-23T03:24:28.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]: dispatch 2024-04-23T03:24:28.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[21886]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]': finished 2024-04-23T03:24:28.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[21886]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T03:24:28.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/44222832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/3928369250' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]: dispatch 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]: dispatch 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[26683]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "afb932fe-d9ff-4c33-b927-0036d4c617fe"}]': finished 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[26683]: osdmap e51: 8 total, 7 up, 8 in 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:28.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:27 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/44222832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-23T03:24:29.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:28 smithi149 ceph-mon[22716]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T03:24:29.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:28 smithi007 ceph-mon[26683]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T03:24:29.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:28 smithi007 ceph-mon[21886]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-23T03:24:31.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:30 smithi149 ceph-mon[22716]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-23T03:24:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:30 smithi007 ceph-mon[26683]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-23T03:24:31.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:30 smithi007 ceph-mon[21886]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-04-23T03:24:33.248 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:32 smithi149 ceph-mon[22716]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T03:24:33.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:32 smithi007 ceph-mon[26683]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T03:24:33.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:32 smithi007 ceph-mon[21886]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-23T03:24:34.251 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:33 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:24:34.251 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:33 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:34.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:33 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:24:34.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:33 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:34.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:33 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:24:34.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:33 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:35.149 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:34 smithi149 ceph-mon[22716]: Deploying daemon osd.7 on smithi149 2024-04-23T03:24:35.149 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:34 smithi149 ceph-mon[22716]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-23T03:24:35.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:34 smithi007 ceph-mon[26683]: Deploying daemon osd.7 on smithi149 2024-04-23T03:24:35.234 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:34 smithi007 ceph-mon[26683]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-23T03:24:35.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:34 smithi007 ceph-mon[21886]: Deploying daemon osd.7 on smithi149 2024-04-23T03:24:35.234 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:34 smithi007 ceph-mon[21886]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 57 KiB/s, 0 objects/s recovering 2024-04-23T03:24:37.250 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:36 smithi149 ceph-mon[22716]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:24:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:36 smithi007 ceph-mon[21886]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:24:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:36 smithi007 ceph-mon[26683]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-23T03:24:39.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:38 smithi149 ceph-mon[22716]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:39.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:39.168 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:39.169 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:39.169 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:38 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[21886]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[26683]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:39.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:38 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:40.166 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:39 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:24:40.166 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:39 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:24:40.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:39 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:24:40.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:39 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:24:40.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:39 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:24:40.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:39 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:24:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:40 smithi149 ceph-mon[22716]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:40 smithi007 ceph-mon[26683]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:40 smithi007 ceph-mon[21886]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:41.370 INFO:teuthology.orchestra.run.smithi149.stdout:Created osd(s) 7 on host 'smithi149' 2024-04-23T03:24:42.000 DEBUG:teuthology.orchestra.run.smithi149:osd.7> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.7.service 2024-04-23T03:24:42.003 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-23T03:24:42.003 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-23T03:24:42.372 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:42 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:42.373 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:42 smithi149 ceph-mon[22716]: from='osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:24:42.373 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:42 smithi149 ceph-mon[22716]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:24:42.373 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:42 smithi149 ceph-mon[22716]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:42.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:42.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[26683]: from='osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:24:42.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[26683]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:24:42.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[26683]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:42.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:42.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[21886]: from='osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:24:42.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[21886]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:24:42.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:42 smithi007 ceph-mon[21886]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:43.644 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[21886]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[21886]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[21886]: from='osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[21886]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[26683]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T03:24:43.645 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[26683]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T03:24:43.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:43.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[26683]: from='osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:43.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[26683]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:43.646 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:43 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:43.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-mon[22716]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T03:24:43.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-mon[22716]: osdmap e52: 8 total, 7 up, 8 in 2024-04-23T03:24:43.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:43.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-mon[22716]: from='osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:43.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-mon[22716]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:24:43.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:43.673 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[34862]: 2024-04-23T03:24:43.381+0000 7f96b0380700 -1 osd.7 0 waiting for initial osdmap 2024-04-23T03:24:43.673 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:24:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[34862]: 2024-04-23T03:24:43.388+0000 7f96abd19700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:24:44.062 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:24:44.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:24:44.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: osdmap e53: 8 total, 7 up, 8 in 2024-04-23T03:24:44.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532] boot 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1765945013' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: osdmap e53: 8 total, 7 up, 8 in 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:44.620 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532] boot 2024-04-23T03:24:44.621 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T03:24:44.621 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.621 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:44 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1765945013' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T03:24:44.642 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1713842683,"num_in_osds":8,"osd_in_since":1713842667,"num_remapped_pgs":1} 2024-04-23T03:24:44.643 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T03:24:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]': finished 2024-04-23T03:24:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: osdmap e53: 8 total, 7 up, 8 in 2024-04-23T03:24:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-23T03:24:44.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: osd.7 [v2:172.21.15.149:6824/2634932532,v1:172.21.15.149:6825/2634932532] boot 2024-04-23T03:24:44.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: osdmap e54: 8 total, 8 up, 8 in 2024-04-23T03:24:44.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:24:44.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:44 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1765945013' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-23T03:24:44.874 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:24:45.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:45 smithi007 ceph-mon[21886]: purged_snaps scrub starts 2024-04-23T03:24:45.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:45 smithi007 ceph-mon[21886]: purged_snaps scrub ok 2024-04-23T03:24:45.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:45 smithi007 ceph-mon[21886]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T03:24:45.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:45 smithi007 ceph-mon[26683]: purged_snaps scrub starts 2024-04-23T03:24:45.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:45 smithi007 ceph-mon[26683]: purged_snaps scrub ok 2024-04-23T03:24:45.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:45 smithi007 ceph-mon[26683]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T03:24:45.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:45 smithi149 ceph-mon[22716]: purged_snaps scrub starts 2024-04-23T03:24:45.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:45 smithi149 ceph-mon[22716]: purged_snaps scrub ok 2024-04-23T03:24:45.624 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:45 smithi149 ceph-mon[22716]: osdmap e55: 8 total, 8 up, 8 in 2024-04-23T03:24:46.622 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:24:46.622 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":56,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","created":"2024-04-23T03:19:44.624487+0000","modified":"2024-04-23T03:24:45.948386+0000","last_up_change":"2024-04-23T03:24:43.941999+0000","last_in_change":"2024-04-23T03:24:27.306027+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T03:22:48.410724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff109592-82df-4023-97fa-b4ea0339f5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6803","nonce":754005596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6805","nonce":754005596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6809","nonce":754005596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6807","nonce":754005596}]},"public_addr":"172.21.15.7:6803/754005596","cluster_addr":"172.21.15.7:6805/754005596","heartbeat_back_addr":"172.21.15.7:6809/754005596","heartbeat_front_addr":"172.21.15.7:6807/754005596","state":["exists","up"]},{"osd":1,"uuid":"b027e9c2-1825-4603-bc6c-922d3ee8bd3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6811","nonce":2742666351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6813","nonce":2742666351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6817","nonce":2742666351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6815","nonce":2742666351}]},"public_addr":"172.21.15.7:6811/2742666351","cluster_addr":"172.21.15.7:6813/2742666351","heartbeat_back_addr":"172.21.15.7:6817/2742666351","heartbeat_front_addr":"172.21.15.7:6815/2742666351","state":["exists","up"]},{"osd":2,"uuid":"4481e007-05de-4368-9251-336f8c16c038","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6819","nonce":1063967933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6821","nonce":1063967933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6825","nonce":1063967933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6823","nonce":1063967933}]},"public_addr":"172.21.15.7:6819/1063967933","cluster_addr":"172.21.15.7:6821/1063967933","heartbeat_back_addr":"172.21.15.7:6825/1063967933","heartbeat_front_addr":"172.21.15.7:6823/1063967933","state":["exists","up"]},{"osd":3,"uuid":"8a4c8dda-e151-41cd-9e1c-e96b64711326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6827","nonce":3422810379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6829","nonce":3422810379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6833","nonce":3422810379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6831","nonce":3422810379}]},"public_addr":"172.21.15.7:6827/3422810379","cluster_addr":"172.21.15.7:6829/3422810379","heartbeat_back_addr":"172.21.15.7:6833/3422810379","heartbeat_front_addr":"172.21.15.7:6831/3422810379","state":["exists","up"]},{"osd":4,"uuid":"7ceeb2e6-0fcf-48e1-995c-10fd315354c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6801","nonce":1331461787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6803","nonce":1331461787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6807","nonce":1331461787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6805","nonce":1331461787}]},"public_addr":"172.21.15.149:6801/1331461787","cluster_addr":"172.21.15.149:6803/1331461787","heartbeat_back_addr":"172.21.15.149:6807/1331461787","heartbeat_front_addr":"172.21.15.149:6805/1331461787","state":["exists","up"]},{"osd":5,"uuid":"ed2557da-b879-4a5d-bcf7-0c291bb47b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6809","nonce":2210178801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6811","nonce":2210178801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6815","nonce":2210178801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6813","nonce":2210178801}]},"public_addr":"172.21.15.149:6809/2210178801","cluster_addr":"172.21.15.149:6811/2210178801","heartbeat_back_addr":"172.21.15.149:6815/2210178801","heartbeat_front_addr":"172.21.15.149:6813/2210178801","state":["exists","up"]},{"osd":6,"uuid":"0c5b5546-4eac-4366-8b18-298c2ece3795","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6817","nonce":3021260633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6819","nonce":3021260633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6823","nonce":3021260633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6821","nonce":3021260633}]},"public_addr":"172.21.15.149:6817/3021260633","cluster_addr":"172.21.15.149:6819/3021260633","heartbeat_back_addr":"172.21.15.149:6823/3021260633","heartbeat_front_addr":"172.21.15.149:6821/3021260633","state":["exists","up"]},{"osd":7,"uuid":"afb932fe-d9ff-4c33-b927-0036d4c617fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6825","nonce":2634932532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6827","nonce":2634932532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6831","nonce":2634932532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6829","nonce":2634932532}]},"public_addr":"172.21.15.149:6825/2634932532","cluster_addr":"172.21.15.149:6827/2634932532","heartbeat_back_addr":"172.21.15.149:6831/2634932532","heartbeat_front_addr":"172.21.15.149:6829/2634932532","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:21:58.876823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:22:22.914943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:22:46.364597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:11.195066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:32.529553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:55.829271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:24:19.317155+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:24:42.606115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.7:0/1785735127":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/752314468":"2024-04-24T03:20:39.280761+0000","172.21.15.7:6801/2136965699":"2024-04-24T03:20:39.280761+0000","172.21.15.7:6800/2136965699":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/3759347266":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/3203492347":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/2924706414":"2024-04-24T03:20:15.204898+0000","172.21.15.7:6801/3542800857":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/860775767":"2024-04-24T03:20:15.204898+0000","172.21.15.7:6800/3542800857":"2024-04-24T03:20:15.204898+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: Detected new or changed devices on smithi149 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 1454M 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:46.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:46 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/551965383' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: Detected new or changed devices on smithi149 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.924 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 1454M 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/551965383' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: Detected new or changed devices on smithi149 2024-04-23T03:24:46.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi149 to 1454M 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: osdmap e56: 8 total, 8 up, 8 in 2024-04-23T03:24:46.926 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:46 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/551965383' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:24:47.322 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-23T03:22:48.410724+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-23T03:24:47.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-23T03:24:47.546 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:24:48.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:47 smithi007 ceph-mon[21886]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T03:24:48.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:47 smithi007 ceph-mon[26683]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T03:24:48.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:47 smithi149 ceph-mon[22716]: osdmap e57: 8 total, 8 up, 8 in 2024-04-23T03:24:49.241 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-23T03:24:49.241 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:48 smithi007 ceph-mon[26683]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T03:24:49.242 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:48 smithi007 ceph-mon[26683]: osdmap e58: 8 total, 8 up, 8 in 2024-04-23T03:24:49.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:48 smithi007 ceph-mon[21886]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T03:24:49.242 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:48 smithi007 ceph-mon[21886]: osdmap e58: 8 total, 8 up, 8 in 2024-04-23T03:24:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:48 smithi149 ceph-mon[22716]: pgmap v171: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-23T03:24:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:48 smithi149 ceph-mon[22716]: osdmap e58: 8 total, 8 up, 8 in 2024-04-23T03:24:49.957 INFO:tasks.cephadm:Adding prometheus.a on smithi149 2024-04-23T03:24:49.957 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi149=a' 2024-04-23T03:24:49.983 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:49 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1696780114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T03:24:50.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:49 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1696780114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T03:24:50.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:49 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1696780114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-23T03:24:51.221 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:50 smithi149 ceph-mon[22716]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-23T03:24:51.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:50 smithi007 ceph-mon[26683]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-23T03:24:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:50 smithi007 ceph-mon[21886]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 103 KiB/s, 0 objects/s recovering 2024-04-23T03:24:51.838 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled prometheus update... 2024-04-23T03:24:52.419 DEBUG:teuthology.orchestra.run.smithi149:prometheus.a> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-23T03:24:52.461 INFO:tasks.cephadm:Adding node-exporter.a on smithi007 2024-04-23T03:24:52.461 INFO:tasks.cephadm:Adding node-exporter.b on smithi149 2024-04-23T03:24:52.461 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi007=a;smithi149=b' 2024-04-23T03:24:52.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:52.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: Saving service prometheus spec with placement smithi149=a;count:1 2024-04-23T03:24:52.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:52.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:52.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:52.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:52.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:52.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-23T03:24:52.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mon[22716]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:24:53.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:53.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: Saving service prometheus spec with placement smithi149=a;count:1 2024-04-23T03:24:53.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:53.172 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[21886]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setuser ceph since I am not root 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setgroup ceph since I am not root 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mgr[22123]: -- 172.21.15.7:0/67620628 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e92eba4340 con 0x55e92eb7cc00 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:53.036+0000 7fe843fcf000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:24:53.173 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:53.170+0000 7fe843fcf000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='client.34310 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: Saving service prometheus spec with placement smithi149=a;count:1 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-23T03:24:53.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:52 smithi007 ceph-mon[26683]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-23T03:24:53.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ignoring --setuser ceph since I am not root 2024-04-23T03:24:53.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ignoring --setgroup ceph since I am not root 2024-04-23T03:24:53.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:52 smithi149 ceph-mgr[23904]: -- 172.21.15.149:0/610432272 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592140b6340 con 0x55921408e800 2024-04-23T03:24:53.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:53.045+0000 7fde565c3000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:24:53.512 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:53.183+0000 7fde565c3000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:24:53.512 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:53.283+0000 7fde565c3000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:24:53.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:53.271+0000 7fe843fcf000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:24:53.812 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:53.599+0000 7fde565c3000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:24:53.812 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:53.684+0000 7fde565c3000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:24:53.862 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:53.589+0000 7fe843fcf000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:24:53.862 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:53.674+0000 7fe843fcf000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:24:54.087 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-mon[22716]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-23T03:24:54.087 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:53 smithi149 ceph-mon[22716]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-23T03:24:54.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-mon[26683]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-23T03:24:54.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-mon[26683]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-23T03:24:54.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-mon[21886]: from='mgr.14152 172.21.15.7:0/3691697171' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-23T03:24:54.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:53 smithi007 ceph-mon[21886]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-23T03:24:55.022 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:54 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:54.850+0000 7fe843fcf000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:24:55.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:54.830+0000 7fde565c3000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:24:55.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:54.993+0000 7fde565c3000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:24:55.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:55.075+0000 7fde565c3000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:24:55.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:55.021+0000 7fe843fcf000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:24:55.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:55.105+0000 7fe843fcf000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:24:55.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:55.388+0000 7fde565c3000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:24:55.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:55.476+0000 7fde565c3000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:24:55.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:55.569+0000 7fde565c3000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:24:55.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:55.434+0000 7fe843fcf000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:24:55.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:55.528+0000 7fe843fcf000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:24:55.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:55.625+0000 7fe843fcf000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:24:56.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:55.813+0000 7fde565c3000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:24:56.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:55 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:55.875+0000 7fe843fcf000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:24:56.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:56.248+0000 7fde565c3000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:24:56.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:56.335+0000 7fe843fcf000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:24:57.116 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:56.734+0000 7fde565c3000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:24:57.116 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:56.825+0000 7fde565c3000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:24:57.116 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:57.115+0000 7fde565c3000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:24:57.247 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:56.839+0000 7fe843fcf000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:24:57.247 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:56.937+0000 7fe843fcf000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:24:57.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:57.203+0000 7fde565c3000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:24:57.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:57.245+0000 7fe843fcf000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:24:57.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:57.339+0000 7fe843fcf000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:24:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:57.608+0000 7fde565c3000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:24:57.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:57.738+0000 7fde565c3000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:24:57.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:57.828+0000 7fde565c3000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:24:58.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:57.757+0000 7fe843fcf000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:24:58.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:57.896+0000 7fe843fcf000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:24:58.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:57.992+0000 7fe843fcf000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:24:58.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:57.940+0000 7fde565c3000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:24:58.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:58.027+0000 7fde565c3000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:24:58.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:58.111+0000 7fe843fcf000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:24:58.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:58.203+0000 7fe843fcf000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:24:58.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:58.566+0000 7fde565c3000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:58.779+0000 7fe843fcf000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[26683]: Standby manager daemon x restarted 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:24:59.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:24:59.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[21886]: Standby manager daemon x restarted 2024-04-23T03:24:59.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[21886]: Standby manager daemon x started 2024-04-23T03:24:59.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:24:59.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:24:59.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:24:59.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:24:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:24:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-mon[22716]: Standby manager daemon x restarted 2024-04-23T03:24:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/1589002548' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:58.932+0000 7fde565c3000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: [23/Apr/2024:03:24:58] ENGINE Bus STARTING 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: CherryPy Checker: 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: The Application mounted at '' has an empty config. 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: [23/Apr/2024:03:24:59] ENGINE Serving on http://:::9283 2024-04-23T03:24:59.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:24:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: [23/Apr/2024:03:24:59] ENGINE Bus STARTED 2024-04-23T03:24:59.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:24:59 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:24:59.155+0000 7fe843fcf000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:25:00.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[21886]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[21886]: Active manager daemon y restarted 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[21886]: Activating manager daemon y 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[21886]: osdmap e59: 8 total, 8 up, 8 in 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[26683]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[26683]: Active manager daemon y restarted 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[26683]: Activating manager daemon y 2024-04-23T03:25:00.268 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-mon[26683]: osdmap e59: 8 total, 8 up, 8 in 2024-04-23T03:25:00.269 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:25:00] ENGINE Bus STARTING 2024-04-23T03:25:00.269 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: CherryPy Checker: 2024-04-23T03:25:00.269 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: The Application mounted at '' has an empty config. 2024-04-23T03:25:00.269 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:25:00.269 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:25:00] ENGINE Serving on http://:::9283 2024-04-23T03:25:00.269 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:25:00] ENGINE Bus STARTED 2024-04-23T03:25:00.331 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:00 smithi149 ceph-mon[22716]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-23T03:25:00.331 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:00 smithi149 ceph-mon[22716]: Active manager daemon y restarted 2024-04-23T03:25:00.331 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:00 smithi149 ceph-mon[22716]: Activating manager daemon y 2024-04-23T03:25:00.331 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:00 smithi149 ceph-mon[22716]: osdmap e59: 8 total, 8 up, 8 in 2024-04-23T03:25:01.024 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled node-exporter update... 2024-04-23T03:25:01.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: mgrmap e18: y(active, starting, since 0.841359s), standbys: x 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:25:01.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: Manager daemon y is now available 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:01.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: mgrmap e18: y(active, starting, since 0.841359s), standbys: x 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:25:01.086 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:25:01.087 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: Manager daemon y is now available 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:25:01.088 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:25:01.107 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: mgrmap e18: y(active, starting, since 0.841359s), standbys: x 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:25:01.108 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: Manager daemon y is now available 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:25:01.109 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:01.110 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:25:01.110 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:25:01.519 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:25:01] ENGINE Bus STARTING 2024-04-23T03:25:01.519 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:25:01] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:25:01.519 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:25:01] ENGINE Bus STARTED 2024-04-23T03:25:01.954 DEBUG:teuthology.orchestra.run.smithi007:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-23T03:25:01.957 DEBUG:teuthology.orchestra.run.smithi149:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-23T03:25:01.959 INFO:tasks.cephadm:Adding alertmanager.a on smithi007 2024-04-23T03:25:01.959 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi007=a' 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: mgrmap e19: y(active, since 1.85614s), standbys: x 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: Saving service node-exporter spec with placement smithi007=a;smithi149=b;count:2 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: [23/Apr/2024:03:25:01] ENGINE Bus STARTING 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: [23/Apr/2024:03:25:01] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: [23/Apr/2024:03:25:01] ENGINE Bus STARTED 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: mgrmap e19: y(active, since 1.85614s), standbys: x 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: Saving service node-exporter spec with placement smithi007=a;smithi149=b;count:2 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: [23/Apr/2024:03:25:01] ENGINE Bus STARTING 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: [23/Apr/2024:03:25:01] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: [23/Apr/2024:03:25:01] ENGINE Bus STARTED 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:02 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: mgrmap e19: y(active, since 1.85614s), standbys: x 2024-04-23T03:25:02.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: Saving service node-exporter spec with placement smithi007=a;smithi149=b;count:2 2024-04-23T03:25:02.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:02.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.226 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: [23/Apr/2024:03:25:01] ENGINE Bus STARTING 2024-04-23T03:25:02.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: [23/Apr/2024:03:25:01] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:25:02.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: [23/Apr/2024:03:25:01] ENGINE Bus STARTED 2024-04-23T03:25:02.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:02.227 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:02 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.146 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:03.146 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-23T03:25:03.146 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.146 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:25:03.146 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:25:03.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.147 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:03 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:25:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-23T03:25:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:25:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: mgrmap e20: y(active, since 2s), standbys: x 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:25:03.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:03.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:03 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:25:04.130 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled alertmanager update... 2024-04-23T03:25:04.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:04 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:25:04.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:04 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:04 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:04 smithi149 ceph-mon[22716]: Deploying daemon prometheus.a on smithi149 2024-04-23T03:25:04.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:04 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[26683]: Deploying daemon prometheus.a on smithi149 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[21886]: Deploying daemon prometheus.a on smithi149 2024-04-23T03:25:04.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:04 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:04.903 DEBUG:teuthology.orchestra.run.smithi007:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-23T03:25:04.906 INFO:tasks.cephadm:Adding grafana.a on smithi149 2024-04-23T03:25:04.906 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi149=a' 2024-04-23T03:25:05.471 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:05 smithi149 ceph-mon[22716]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:05.471 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:05 smithi149 ceph-mon[22716]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:25:05.471 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:05 smithi149 ceph-mon[22716]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-04-23T03:25:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:05 smithi007 ceph-mon[26683]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:05 smithi007 ceph-mon[26683]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:25:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:05 smithi007 ceph-mon[26683]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-04-23T03:25:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:05 smithi007 ceph-mon[21886]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:05 smithi007 ceph-mon[21886]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:25:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:05 smithi007 ceph-mon[21886]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-04-23T03:25:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:07 smithi007 ceph-mon[21886]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:07 smithi007 ceph-mon[26683]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:07.597 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:07 smithi149 ceph-mon[22716]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:07.752 INFO:teuthology.orchestra.run.smithi149.stdout:Scheduled grafana update... 2024-04-23T03:25:08.754 DEBUG:teuthology.orchestra.run.smithi149:grafana.a> sudo journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@grafana.a.service 2024-04-23T03:25:08.757 INFO:tasks.cephadm:Setting up client nodes... 2024-04-23T03:25:08.757 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T03:25:09.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:08 smithi007 ceph-mon[26683]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:25:09.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:08 smithi007 ceph-mon[26683]: Saving service grafana spec with placement smithi149=a;count:1 2024-04-23T03:25:09.077 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:08 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:09.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:08 smithi007 ceph-mon[21886]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:25:09.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:08 smithi007 ceph-mon[21886]: Saving service grafana spec with placement smithi149=a;count:1 2024-04-23T03:25:09.078 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:08 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:09.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:08 smithi149 ceph-mon[22716]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi149=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:25:09.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:08 smithi149 ceph-mon[22716]: Saving service grafana spec with placement smithi149=a;count:1 2024-04-23T03:25:09.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:08 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:10.035 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:09 smithi007 ceph-mon[26683]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:10.035 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:09 smithi007 ceph-mon[21886]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:10.048 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:09 smithi149 ceph-mon[22716]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:10.821 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-04-23T03:25:10.821 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQAWKidm0SKxMBAAtugmYDR5bZR0fZzhwLg3sA== 2024-04-23T03:25:11.435 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-23T03:25:11.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-23T03:25:11.435 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-23T03:25:11.473 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-23T03:25:12.073 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:11 smithi149 ceph-mon[22716]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:12.073 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:11 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3309273076' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:12.074 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:11 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:12.074 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:11 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[26683]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3309273076' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[21886]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3309273076' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:12.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:11 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T03:25:13.971 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:13 smithi149 ceph-mon[22716]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:14.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:13 smithi007 ceph-mon[26683]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:14.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:13 smithi007 ceph-mon[21886]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:14.259 INFO:teuthology.orchestra.run.smithi149.stdout:[client.1] 2024-04-23T03:25:14.260 INFO:teuthology.orchestra.run.smithi149.stdout: key = AQAaKidmiiQ6DxAAwgjvHt5AU1+Igti6AXhOlg== 2024-04-23T03:25:15.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:14 smithi149 ceph-mon[22716]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:15.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:14 smithi149 ceph-mon[22716]: from='client.? 172.21.15.149:0/72147229' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:15.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:14 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:15.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:14 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[26683]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[26683]: from='client.? 172.21.15.149:0/72147229' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[21886]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[21886]: from='client.? 172.21.15.149:0/72147229' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-23T03:25:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:14 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-23T03:25:16.267 DEBUG:teuthology.orchestra.run.smithi149:> set -ex 2024-04-23T03:25:16.267 DEBUG:teuthology.orchestra.run.smithi149:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-23T03:25:16.267 DEBUG:teuthology.orchestra.run.smithi149:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-23T03:25:16.302 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-23T03:25:16.302 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-23T03:25:16.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-23T03:25:16.530 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:16.747 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:25:16.746Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-23T03:25:16.747 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:25:16.746Z caller=main.go:947 level=info msg="TSDB started" 2024-04-23T03:25:16.748 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:25:16.746Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-23T03:25:17.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:25:16.762Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.361052ms db_storage=757ns remote_storage=1.384µs web_handler=364ns query_engine=797ns scrape=340.816µs scrape_sd=21.933µs notify=843ns notify_sd=1.504µs rules=14.543228ms 2024-04-23T03:25:17.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:25:16.762Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-23T03:25:17.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 bash[37857]: 167360a37db66824419f004b96933e2577527893fa9bc30cc9a183b7d0b42bc2 2024-04-23T03:25:17.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:25:16 smithi149 systemd[1]: Started Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:25:18.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:17 smithi007 ceph-mon[26683]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:18.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:17 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:18.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:17 smithi007 ceph-mon[26683]: Deploying daemon node-exporter.a on smithi007 2024-04-23T03:25:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:17 smithi007 ceph-mon[21886]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:17 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:18.075 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:17 smithi007 ceph-mon[21886]: Deploying daemon node-exporter.a on smithi007 2024-04-23T03:25:18.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:17 smithi149 ceph-mon[22716]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:18.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:17 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:18.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:17 smithi149 ceph-mon[22716]: Deploying daemon node-exporter.a on smithi007 2024-04-23T03:25:18.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:25:18.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:18 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2036791360' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T03:25:18.951 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:18 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2036791360' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T03:25:18.974 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":20,"active_gid":24343,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":3848738408},{"type":"v1","addr":"172.21.15.7:6801","nonce":3848738408}]},"active_addr":"172.21.15.7:6801/3848738408","active_change":"2024-04-23T03:24:59.158856+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34322,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/","prometheus":"http://172.21.15.7:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":59,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":782590966}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":3858595978}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":541957860}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2317307583}]}]}} 2024-04-23T03:25:18.980 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-23T03:25:18.980 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-23T03:25:18.980 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T03:25:19.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:18 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2036791360' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-23T03:25:19.231 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:19.333 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:19 smithi007 bash[41493]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-23T03:25:19.630 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:19 smithi007 bash[41493]: Getting image source signatures 2024-04-23T03:25:19.630 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:19 smithi007 bash[41493]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-23T03:25:19.630 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:19 smithi007 bash[41493]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-23T03:25:19.630 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:19 smithi007 bash[41493]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-23T03:25:20.036 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:19 smithi007 ceph-mon[26683]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:20.037 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:19 smithi007 ceph-mon[21886]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:20.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:19 smithi149 ceph-mon[22716]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:21.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:21 smithi007 ceph-mon[21886]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:21.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:21 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:21.321 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:21 smithi007 ceph-mon[26683]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:21.322 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:21 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:21.322 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:21 smithi007 bash[41493]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-23T03:25:21.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:21 smithi149 ceph-mon[22716]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:21.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:21 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:21.583 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:21 smithi007 bash[41493]: Writing manifest to image destination 2024-04-23T03:25:21.797 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:25:21.797 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","created":"2024-04-23T03:19:44.624487+0000","modified":"2024-04-23T03:24:59.158436+0000","last_up_change":"2024-04-23T03:24:43.941999+0000","last_in_change":"2024-04-23T03:24:27.306027+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T03:22:48.410724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff109592-82df-4023-97fa-b4ea0339f5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6803","nonce":754005596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6805","nonce":754005596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6809","nonce":754005596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6807","nonce":754005596}]},"public_addr":"172.21.15.7:6803/754005596","cluster_addr":"172.21.15.7:6805/754005596","heartbeat_back_addr":"172.21.15.7:6809/754005596","heartbeat_front_addr":"172.21.15.7:6807/754005596","state":["exists","up"]},{"osd":1,"uuid":"b027e9c2-1825-4603-bc6c-922d3ee8bd3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6811","nonce":2742666351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6813","nonce":2742666351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6817","nonce":2742666351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6815","nonce":2742666351}]},"public_addr":"172.21.15.7:6811/2742666351","cluster_addr":"172.21.15.7:6813/2742666351","heartbeat_back_addr":"172.21.15.7:6817/2742666351","heartbeat_front_addr":"172.21.15.7:6815/2742666351","state":["exists","up"]},{"osd":2,"uuid":"4481e007-05de-4368-9251-336f8c16c038","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6819","nonce":1063967933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6821","nonce":1063967933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6825","nonce":1063967933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6823","nonce":1063967933}]},"public_addr":"172.21.15.7:6819/1063967933","cluster_addr":"172.21.15.7:6821/1063967933","heartbeat_back_addr":"172.21.15.7:6825/1063967933","heartbeat_front_addr":"172.21.15.7:6823/1063967933","state":["exists","up"]},{"osd":3,"uuid":"8a4c8dda-e151-41cd-9e1c-e96b64711326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6827","nonce":3422810379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6829","nonce":3422810379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6833","nonce":3422810379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6831","nonce":3422810379}]},"public_addr":"172.21.15.7:6827/3422810379","cluster_addr":"172.21.15.7:6829/3422810379","heartbeat_back_addr":"172.21.15.7:6833/3422810379","heartbeat_front_addr":"172.21.15.7:6831/3422810379","state":["exists","up"]},{"osd":4,"uuid":"7ceeb2e6-0fcf-48e1-995c-10fd315354c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6801","nonce":1331461787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6803","nonce":1331461787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6807","nonce":1331461787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6805","nonce":1331461787}]},"public_addr":"172.21.15.149:6801/1331461787","cluster_addr":"172.21.15.149:6803/1331461787","heartbeat_back_addr":"172.21.15.149:6807/1331461787","heartbeat_front_addr":"172.21.15.149:6805/1331461787","state":["exists","up"]},{"osd":5,"uuid":"ed2557da-b879-4a5d-bcf7-0c291bb47b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6809","nonce":2210178801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6811","nonce":2210178801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6815","nonce":2210178801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6813","nonce":2210178801}]},"public_addr":"172.21.15.149:6809/2210178801","cluster_addr":"172.21.15.149:6811/2210178801","heartbeat_back_addr":"172.21.15.149:6815/2210178801","heartbeat_front_addr":"172.21.15.149:6813/2210178801","state":["exists","up"]},{"osd":6,"uuid":"0c5b5546-4eac-4366-8b18-298c2ece3795","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6817","nonce":3021260633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6819","nonce":3021260633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6823","nonce":3021260633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6821","nonce":3021260633}]},"public_addr":"172.21.15.149:6817/3021260633","cluster_addr":"172.21.15.149:6819/3021260633","heartbeat_back_addr":"172.21.15.149:6823/3021260633","heartbeat_front_addr":"172.21.15.149:6821/3021260633","state":["exists","up"]},{"osd":7,"uuid":"afb932fe-d9ff-4c33-b927-0036d4c617fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6825","nonce":2634932532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6827","nonce":2634932532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6831","nonce":2634932532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6829","nonce":2634932532}]},"public_addr":"172.21.15.149:6825/2634932532","cluster_addr":"172.21.15.149:6827/2634932532","heartbeat_back_addr":"172.21.15.149:6831/2634932532","heartbeat_front_addr":"172.21.15.149:6829/2634932532","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:21:58.876823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:22:22.914943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:22:46.364597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:11.195066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:32.529553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:55.829271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:24:19.317155+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:24:42.606115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1711557554":"2024-04-24T03:24:59.158388+0000","172.21.15.7:0/136969602":"2024-04-24T03:24:59.158388+0000","172.21.15.7:0/2530963392":"2024-04-24T03:24:59.158388+0000","172.21.15.7:6801/3100710170":"2024-04-24T03:24:59.158388+0000","172.21.15.7:6800/3100710170":"2024-04-24T03:24:59.158388+0000","172.21.15.7:0/1785735127":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/752314468":"2024-04-24T03:20:39.280761+0000","172.21.15.7:6801/2136965699":"2024-04-24T03:20:39.280761+0000","172.21.15.7:6800/2136965699":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/3759347266":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/3203492347":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/2924706414":"2024-04-24T03:20:15.204898+0000","172.21.15.7:6801/3542800857":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/860775767":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/2507582309":"2024-04-24T03:24:59.158388+0000","172.21.15.7:6800/3542800857":"2024-04-24T03:20:15.204898+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T03:25:22.062 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:21 smithi007 podman[41493]: 2024-04-23 03:25:21.838324886 +0000 UTC m=+2.772867228 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-23T03:25:22.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/150907257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:25:22.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/150907257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:25:22.334 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 podman[41493]: 2024-04-23 03:25:22.12139527 +0000 UTC m=+3.055937639 container create e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:25:22.334 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 podman[41493]: 2024-04-23 03:25:22.290129319 +0000 UTC m=+3.224671649 container init e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:25:22.334 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 podman[41493]: 2024-04-23 03:25:22.292985743 +0000 UTC m=+3.227528071 container start e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:25:22.334 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.297Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.297Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.297Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.297Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=arp 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-23T03:25:22.335 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=edac 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-23T03:25:22.336 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=os 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=stat 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-23T03:25:22.337 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=time 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=uname 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-23T03:25:22.338 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[41667]: ts=2024-04-23T03:25:22.298Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-23T03:25:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:22 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/150907257' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:25:22.833 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 bash[41493]: e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b 2024-04-23T03:25:22.833 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:25:22 smithi007 systemd[1]: Started Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:25:22.901 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-23T03:25:22.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-23T03:25:23.125 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:23.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:23 smithi149 bash[38096]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-23T03:25:23.548 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:23 smithi007 ceph-mon[21886]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:23.549 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:23 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:23.549 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:23 smithi007 ceph-mon[21886]: Deploying daemon node-exporter.b on smithi149 2024-04-23T03:25:23.549 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:23 smithi007 ceph-mon[26683]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:23.549 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:23 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:23.549 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:23 smithi007 ceph-mon[26683]: Deploying daemon node-exporter.b on smithi149 2024-04-23T03:25:23.682 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:23 smithi149 ceph-mon[22716]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:23.682 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:23 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:23.682 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:23 smithi149 ceph-mon[22716]: Deploying daemon node-exporter.b on smithi149 2024-04-23T03:25:24.172 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:23 smithi149 bash[38096]: Getting image source signatures 2024-04-23T03:25:24.173 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:23 smithi149 bash[38096]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-23T03:25:24.173 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:23 smithi149 bash[38096]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-23T03:25:24.173 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:23 smithi149 bash[38096]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-23T03:25:24.824 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:25:24.825 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":59,"fsid":"3d05926e-0120-11ef-bc93-c7b262605968","created":"2024-04-23T03:19:44.624487+0000","modified":"2024-04-23T03:24:59.158436+0000","last_up_change":"2024-04-23T03:24:43.941999+0000","last_in_change":"2024-04-23T03:24:27.306027+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-23T03:22:48.410724+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"ff109592-82df-4023-97fa-b4ea0339f5fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6803","nonce":754005596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6805","nonce":754005596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6809","nonce":754005596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":754005596},{"type":"v1","addr":"172.21.15.7:6807","nonce":754005596}]},"public_addr":"172.21.15.7:6803/754005596","cluster_addr":"172.21.15.7:6805/754005596","heartbeat_back_addr":"172.21.15.7:6809/754005596","heartbeat_front_addr":"172.21.15.7:6807/754005596","state":["exists","up"]},{"osd":1,"uuid":"b027e9c2-1825-4603-bc6c-922d3ee8bd3a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6811","nonce":2742666351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6813","nonce":2742666351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6817","nonce":2742666351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":2742666351},{"type":"v1","addr":"172.21.15.7:6815","nonce":2742666351}]},"public_addr":"172.21.15.7:6811/2742666351","cluster_addr":"172.21.15.7:6813/2742666351","heartbeat_back_addr":"172.21.15.7:6817/2742666351","heartbeat_front_addr":"172.21.15.7:6815/2742666351","state":["exists","up"]},{"osd":2,"uuid":"4481e007-05de-4368-9251-336f8c16c038","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6819","nonce":1063967933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6821","nonce":1063967933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6825","nonce":1063967933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":1063967933},{"type":"v1","addr":"172.21.15.7:6823","nonce":1063967933}]},"public_addr":"172.21.15.7:6819/1063967933","cluster_addr":"172.21.15.7:6821/1063967933","heartbeat_back_addr":"172.21.15.7:6825/1063967933","heartbeat_front_addr":"172.21.15.7:6823/1063967933","state":["exists","up"]},{"osd":3,"uuid":"8a4c8dda-e151-41cd-9e1c-e96b64711326","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6827","nonce":3422810379}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6829","nonce":3422810379}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6833","nonce":3422810379}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":3422810379},{"type":"v1","addr":"172.21.15.7:6831","nonce":3422810379}]},"public_addr":"172.21.15.7:6827/3422810379","cluster_addr":"172.21.15.7:6829/3422810379","heartbeat_back_addr":"172.21.15.7:6833/3422810379","heartbeat_front_addr":"172.21.15.7:6831/3422810379","state":["exists","up"]},{"osd":4,"uuid":"7ceeb2e6-0fcf-48e1-995c-10fd315354c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6800","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6801","nonce":1331461787}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6802","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6803","nonce":1331461787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6806","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6807","nonce":1331461787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6804","nonce":1331461787},{"type":"v1","addr":"172.21.15.149:6805","nonce":1331461787}]},"public_addr":"172.21.15.149:6801/1331461787","cluster_addr":"172.21.15.149:6803/1331461787","heartbeat_back_addr":"172.21.15.149:6807/1331461787","heartbeat_front_addr":"172.21.15.149:6805/1331461787","state":["exists","up"]},{"osd":5,"uuid":"ed2557da-b879-4a5d-bcf7-0c291bb47b74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6808","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6809","nonce":2210178801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6810","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6811","nonce":2210178801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6814","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6815","nonce":2210178801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6812","nonce":2210178801},{"type":"v1","addr":"172.21.15.149:6813","nonce":2210178801}]},"public_addr":"172.21.15.149:6809/2210178801","cluster_addr":"172.21.15.149:6811/2210178801","heartbeat_back_addr":"172.21.15.149:6815/2210178801","heartbeat_front_addr":"172.21.15.149:6813/2210178801","state":["exists","up"]},{"osd":6,"uuid":"0c5b5546-4eac-4366-8b18-298c2ece3795","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6816","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6817","nonce":3021260633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6818","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6819","nonce":3021260633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6822","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6823","nonce":3021260633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6820","nonce":3021260633},{"type":"v1","addr":"172.21.15.149:6821","nonce":3021260633}]},"public_addr":"172.21.15.149:6817/3021260633","cluster_addr":"172.21.15.149:6819/3021260633","heartbeat_back_addr":"172.21.15.149:6823/3021260633","heartbeat_front_addr":"172.21.15.149:6821/3021260633","state":["exists","up"]},{"osd":7,"uuid":"afb932fe-d9ff-4c33-b927-0036d4c617fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6824","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6825","nonce":2634932532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6826","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6827","nonce":2634932532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6830","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6831","nonce":2634932532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.149:6828","nonce":2634932532},{"type":"v1","addr":"172.21.15.149:6829","nonce":2634932532}]},"public_addr":"172.21.15.149:6825/2634932532","cluster_addr":"172.21.15.149:6827/2634932532","heartbeat_back_addr":"172.21.15.149:6831/2634932532","heartbeat_front_addr":"172.21.15.149:6829/2634932532","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:21:58.876823+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:22:22.914943+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:22:46.364597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:11.195066+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:32.529553+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:23:55.829271+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:24:19.317155+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-23T03:24:42.606115+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/1711557554":"2024-04-24T03:24:59.158388+0000","172.21.15.7:0/136969602":"2024-04-24T03:24:59.158388+0000","172.21.15.7:0/2530963392":"2024-04-24T03:24:59.158388+0000","172.21.15.7:6801/3100710170":"2024-04-24T03:24:59.158388+0000","172.21.15.7:6800/3100710170":"2024-04-24T03:24:59.158388+0000","172.21.15.7:0/1785735127":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/752314468":"2024-04-24T03:20:39.280761+0000","172.21.15.7:6801/2136965699":"2024-04-24T03:20:39.280761+0000","172.21.15.7:6800/2136965699":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/3759347266":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/3203492347":"2024-04-24T03:20:39.280761+0000","172.21.15.7:0/2924706414":"2024-04-24T03:20:15.204898+0000","172.21.15.7:6801/3542800857":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/860775767":"2024-04-24T03:20:15.204898+0000","172.21.15.7:0/2507582309":"2024-04-24T03:24:59.158388+0000","172.21.15.7:6800/3542800857":"2024-04-24T03:20:15.204898+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-23T03:25:25.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:24 smithi007 ceph-mon[21886]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:25.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:24 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2921780501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:25:25.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:24 smithi007 ceph-mon[26683]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:25.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:24 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2921780501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:25:25.217 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:24 smithi149 ceph-mon[22716]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:25.217 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:24 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2921780501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-23T03:25:25.217 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 bash[38096]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-23T03:25:25.218 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 bash[38096]: Writing manifest to image destination 2024-04-23T03:25:25.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-23T03:25:25.580 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-23T03:25:25.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-23T03:25:25.581 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-23T03:25:25.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-23T03:25:25.582 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-23T03:25:25.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-23T03:25:25.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 podman[38096]: 2024-04-23 03:25:25.587343911 +0000 UTC m=+2.420524836 container create 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 podman[38096]: 2024-04-23 03:25:25.497099839 +0000 UTC m=+2.330280710 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 podman[38096]: 2024-04-23 03:25:25.781091027 +0000 UTC m=+2.614271887 container init 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 podman[38096]: 2024-04-23 03:25:25.783920632 +0000 UTC m=+2.617101496 container start 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.788Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.788Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-23T03:25:25.858 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=arp 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-23T03:25:25.859 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=edac 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-23T03:25:25.860 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=os 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-23T03:25:25.861 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=stat 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=time 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=uname 2024-04-23T03:25:25.862 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-23T03:25:25.863 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-23T03:25:25.863 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-23T03:25:25.863 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.789Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-23T03:25:25.863 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[38146]: ts=2024-04-23T03:25:25.790Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-23T03:25:25.863 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 bash[38096]: 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed 2024-04-23T03:25:26.050 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.051 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.055 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.055 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.059 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.059 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.060 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.062 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:26.172 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:25:25 smithi149 systemd[1]: Started Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:25:27.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:27.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:27.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:25:27.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:25:27.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[26683]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:27.057 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:27.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:27.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:25:27.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:25:27.058 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:26 smithi007 ceph-mon[21886]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:27.278 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:26 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:27.279 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:26 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:25:27.279 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:26 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:25:27.279 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:26 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:25:27.279 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:26 smithi149 ceph-mon[22716]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:27.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:25:27 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:25:29.559 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:29 smithi007 ceph-mon[26683]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:29.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:29 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:29.560 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:29 smithi007 ceph-mon[21886]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:29.560 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:29 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:29 smithi149 ceph-mon[22716]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:29.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:29 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:30.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:30] "GET /metrics HTTP/1.1" 200 191474 "" "Prometheus/2.33.4" 2024-04-23T03:25:31.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:31 smithi007 ceph-mon[26683]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:31.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:31 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:31.320 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:31 smithi007 ceph-mon[21886]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:31.321 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:31 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:31.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:31 smithi149 ceph-mon[22716]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:31 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:33.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:33 smithi007 ceph-mon[26683]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:33.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:33 smithi007 ceph-mon[21886]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:33.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:33 smithi149 ceph-mon[22716]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:35.103 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:34 smithi007 ceph-mon[26683]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:35.103 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:34 smithi007 ceph-mon[21886]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:34 smithi149 ceph-mon[22716]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:36.355 INFO:teuthology.orchestra.run.smithi007.stdout:77309411363 2024-04-23T03:25:36.356 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-23T03:25:37.184 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:37 smithi007 ceph-mon[26683]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:37.184 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:37 smithi007 ceph-mon[21886]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:37 smithi149 ceph-mon[22716]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:37.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:25:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:25:38.601 INFO:teuthology.orchestra.run.smithi007.stdout:231928233996 2024-04-23T03:25:38.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-23T03:25:39.205 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:39 smithi007 ceph-mon[21886]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:39.205 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:39 smithi007 ceph-mon[26683]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:39 smithi149 ceph-mon[22716]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:40.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:40] "GET /metrics HTTP/1.1" 200 191474 "" "Prometheus/2.33.4" 2024-04-23T03:25:41.309 INFO:teuthology.orchestra.run.smithi007.stdout:128849018905 2024-04-23T03:25:41.310 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-23T03:25:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:41 smithi007 ceph-mon[26683]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:41 smithi007 ceph-mon[21886]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:41 smithi149 ceph-mon[22716]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:41.542 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:41.543 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:41.929 INFO:teuthology.orchestra.run.smithi007.stdout:55834574888 2024-04-23T03:25:41.930 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-23T03:25:42.692 INFO:teuthology.orchestra.run.smithi007.stdout:197568495634 2024-04-23T03:25:42.692 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-23T03:25:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:43 smithi007 ceph-mon[21886]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:43.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:43 smithi007 ceph-mon[26683]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:43.412 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:43.414 INFO:teuthology.orchestra.run.smithi007.stdout:107374182430 2024-04-23T03:25:43.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-23T03:25:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:43 smithi149 ceph-mon[22716]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:43.860 INFO:teuthology.orchestra.run.smithi007.stdout:163208757269 2024-04-23T03:25:43.861 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-23T03:25:44.314 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:44.496 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:45.277 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:44 smithi007 ceph-mon[26683]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:45.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:44 smithi007 ceph-mon[21886]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:44 smithi149 ceph-mon[22716]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:47.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:47 smithi007 ceph-mon[26683]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:47.317 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:47 smithi007 ceph-mon[21886]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:47 smithi149 ceph-mon[22716]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:47.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:25:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:25:48.821 INFO:teuthology.orchestra.run.smithi007.stdout:34359738415 2024-04-23T03:25:48.822 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-23T03:25:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:49 smithi007 ceph-mon[26683]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:49.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:49 smithi007 ceph-mon[21886]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:49.417 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:49 smithi149 ceph-mon[22716]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:49.430 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:50.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:50.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:50.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-mon[26683]: Deploying daemon alertmanager.a on smithi007 2024-04-23T03:25:50.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:50.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:50.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-mon[21886]: Deploying daemon alertmanager.a on smithi007 2024-04-23T03:25:50.743 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:25:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:50] "GET /metrics HTTP/1.1" 200 191459 "" "Prometheus/2.33.4" 2024-04-23T03:25:50.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:50 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:50.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:50 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:25:50.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:50 smithi149 ceph-mon[22716]: Deploying daemon alertmanager.a on smithi007 2024-04-23T03:25:50.948 INFO:teuthology.orchestra.run.smithi007.stdout:197568495636 2024-04-23T03:25:51.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:51 smithi007 ceph-mon[26683]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:51.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:51 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2976541698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T03:25:51.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:51 smithi007 ceph-mon[21886]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:51.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:51 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2976541698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T03:25:51.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:51 smithi149 ceph-mon[22716]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:51.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:51 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2976541698' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-23T03:25:53.393 INFO:teuthology.orchestra.run.smithi007.stdout:231928233999 2024-04-23T03:25:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:53 smithi007 ceph-mon[21886]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:53 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2871322924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T03:25:53.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:53 smithi007 ceph-mon[26683]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:53.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:53 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2871322924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T03:25:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:53 smithi149 ceph-mon[22716]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:53 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2871322924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-23T03:25:54.947 INFO:teuthology.orchestra.run.smithi007.stdout:128849018908 2024-04-23T03:25:55.024 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:54 smithi007 ceph-mon[26683]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:55.024 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:54 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1097274541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T03:25:55.024 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:54 smithi007 ceph-mon[21886]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:55.024 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:54 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1097274541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T03:25:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:54 smithi149 ceph-mon[22716]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:54 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1097274541' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-23T03:25:55.826 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495634 got 197568495636 for osd.6 2024-04-23T03:25:55.826 DEBUG:teuthology.parallel:result is None 2024-04-23T03:25:56.236 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233996 got 231928233999 for osd.7 2024-04-23T03:25:56.236 DEBUG:teuthology.parallel:result is None 2024-04-23T03:25:57.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:57 smithi007 ceph-mon[26683]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:57.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:57 smithi007 ceph-mon[21886]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:57 smithi149 ceph-mon[22716]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:57.917 INFO:teuthology.orchestra.run.smithi007.stdout:163208757272 2024-04-23T03:25:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:25:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:25:58.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:58 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1970334725' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T03:25:58.290 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:58 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1970334725' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T03:25:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:58 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1970334725' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-23T03:25:58.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757269 got 163208757272 for osd.5 2024-04-23T03:25:58.805 DEBUG:teuthology.parallel:result is None 2024-04-23T03:25:59.196 INFO:teuthology.orchestra.run.smithi007.stdout:77309411368 2024-04-23T03:25:59.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:25:59 smithi007 ceph-mon[26683]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:59.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:25:59 smithi007 ceph-mon[21886]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:25:59 smithi149 ceph-mon[22716]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:25:59.775 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018905 got 128849018908 for osd.4 2024-04-23T03:25:59.775 DEBUG:teuthology.parallel:result is None 2024-04-23T03:25:59.776 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:25:59.834 INFO:teuthology.orchestra.run.smithi007.stdout:107374182434 2024-04-23T03:26:00.169 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:00 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2341426706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T03:26:00.169 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:00 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1815690778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T03:26:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:00 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2341426706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T03:26:00.169 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:00 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1815690778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T03:26:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:00 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2341426706' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-23T03:26:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:00 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1815690778' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-23T03:26:00.777 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:26:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:00] "GET /metrics HTTP/1.1" 200 191460 "" "Prometheus/2.33.4" 2024-04-23T03:26:01.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[26683]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/501525115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[21886]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:26:01.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:01 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/501525115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T03:26:01.357 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182430 got 107374182434 for osd.3 2024-04-23T03:26:01.357 DEBUG:teuthology.parallel:result is None 2024-04-23T03:26:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:01 smithi149 ceph-mon[22716]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:26:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:26:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:01 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/501525115' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-23T03:26:01.938 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411368 for osd.2 2024-04-23T03:26:01.939 DEBUG:teuthology.parallel:result is None 2024-04-23T03:26:01.984 INFO:teuthology.orchestra.run.smithi007.stdout:34359738418 2024-04-23T03:26:02.726 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738415 got 34359738418 for osd.0 2024-04-23T03:26:02.726 DEBUG:teuthology.parallel:result is None 2024-04-23T03:26:03.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:03 smithi007 ceph-mon[21886]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:03.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:03 smithi007 ceph-mon[26683]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:03 smithi149 ceph-mon[22716]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:55834574893 2024-04-23T03:26:05.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:04 smithi007 ceph-mon[26683]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:05.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:04 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2073544181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T03:26:05.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:04 smithi007 ceph-mon[21886]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:05.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:04 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2073544181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T03:26:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:04 smithi149 ceph-mon[22716]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:04 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2073544181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-23T03:26:06.072 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574888 got 55834574893 for osd.1 2024-04-23T03:26:06.072 DEBUG:teuthology.parallel:result is None 2024-04-23T03:26:06.072 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-23T03:26:06.073 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T03:26:07.116 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:26:07.269 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:07 smithi007 ceph-mon[26683]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:07.270 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:07 smithi007 ceph-mon[21886]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:07 smithi149 ceph-mon[22716]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:07.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:26:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:26:08.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[26683]: Deploying daemon grafana.a on smithi149 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:26:08.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:26:08.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:08 smithi007 ceph-mon[21886]: Deploying daemon grafana.a on smithi149 2024-04-23T03:26:08.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:26:08.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:26:08.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:08.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:08 smithi149 ceph-mon[22716]: Deploying daemon grafana.a on smithi149 2024-04-23T03:26:08.935 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:26:08.935 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-23T03:26:09.573 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:09 smithi149 ceph-mon[22716]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:09.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:09 smithi007 ceph-mon[26683]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:09 smithi007 ceph-mon[21886]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:09.619 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-23T03:26:08.018427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48820,"kb_used_data":6068,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912396,"statfs":{"total":767960285184,"available":767910293504,"internally_reserved":0,"allocated":6213632,"data_stored":3487074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002221"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-23T03:25:00.092995+0000","last_change":"2024-04-23T03:24:47.959672+0000","last_active":"2024-04-23T03:25:00.092995+0000","last_peered":"2024-04-23T03:25:00.092995+0000","last_clean":"2024-04-23T03:25:00.092995+0000","last_became_active":"2024-04-23T03:24:47.959296+0000","last_became_peered":"2024-04-23T03:24:47.959296+0000","last_unstale":"2024-04-23T03:25:00.092995+0000","last_undegraded":"2024-04-23T03:25:00.092995+0000","last_fullsized":"2024-04-23T03:25:00.092995+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T03:22:48.947740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T03:22:48.947740+0000","last_clean_scrub_stamp":"2024-04-23T03:22:48.947740+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:43:59.817136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.495}]},{"osd":2,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39500000000000002}]},{"osd":5,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.56799999999999995}]},{"osd":6,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55200000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.748}]},{"osd":2,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71599999999999997}]},{"osd":4,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.624}]},{"osd":5,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.55000000000000004}]},{"osd":7,"last update":"Tue Apr 23 03:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6328,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738824,"statfs":{"total":95995035648,"available":95988555776,"internally_reserved":0,"allocated":573440,"data_stored":237332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.35899999999999999,"15min":0.34300000000000003},"min":{"1min":0.27300000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.38700000000000001,"15min":0.36299999999999999},"min":{"1min":0.30299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46100000000000002}]},{"osd":2,"last update":"Tue Apr 23 03:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40699999999999997,"15min":0.39700000000000002},"min":{"1min":0.29799999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.58299999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.42199999999999999,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.628,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Tue Apr 23 03:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47599999999999998,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.47299999999999998,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Apr 23 03:25:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.629},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Apr 23 03:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.629},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.71799999999999997,"15min":0.65800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":10.391,"5min":10.391,"15min":10.391},"last":0.70299999999999996}]},{"osd":6,"last update":"Tue Apr 23 03:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.64500000000000002}]},{"osd":7,"last update":"Tue Apr 23 03:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.67800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 23 03:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.40699999999999997,"15min":0.38100000000000001},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35199999999999998}]},{"osd":2,"last update":"Tue Apr 23 03:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.437,"15min":0.42699999999999999},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36199999999999999}]},{"osd":3,"last update":"Tue Apr 23 03:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.45200000000000001,"15min":0.45900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58999999999999997,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.622,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.33300000000000002}]},{"osd":4,"last update":"Tue Apr 23 03:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83299999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.80600000000000005,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Apr 23 03:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84299999999999997,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.749}]},{"osd":6,"last update":"Tue Apr 23 03:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.78600000000000003}]},{"osd":7,"last update":"Tue Apr 23 03:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55300000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39000000000000001,"15min":0.376},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Apr 23 03:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.40899999999999997,"15min":0.39600000000000002},"min":{"1min":0.30599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.41199999999999998},"min":{"1min":0.29299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.307}]},{"osd":3,"last update":"Tue Apr 23 03:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Apr 23 03:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89100000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.502}]},{"osd":5,"last update":"Tue Apr 23 03:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83699999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63400000000000001,"15min":0.64000000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.81399999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.42599999999999999}]},{"osd":6,"last update":"Tue Apr 23 03:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48399999999999999}]},{"osd":7,"last update":"Tue Apr 23 03:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.63400000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.433,"15min":0.435},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Apr 23 03:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.50800000000000001}]},{"osd":2,"last update":"Tue Apr 23 03:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46899999999999997,"15min":0.46999999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.59899999999999998,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70599999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.97599999999999998}]},{"osd":4,"last update":"Tue Apr 23 03:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.80100000000000005}]},{"osd":5,"last update":"Tue Apr 23 03:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.625,"15min":0.626},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Apr 23 03:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.81499999999999995}]},{"osd":7,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.91900000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.60199999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.70299999999999996,"5min":0.997,"15min":0.997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74099999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68799999999999994}]},{"osd":1,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75900000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.60299999999999998,"15min":0.61899999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.105,"15min":1.105},"last":0.56200000000000006}]},{"osd":2,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71699999999999997,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.69099999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78000000000000003,"5min":0.874,"15min":0.874},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.55400000000000005}]},{"osd":5,"last update":"Tue Apr 23 03:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.32400000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.52800000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40799999999999997,"15min":0.40100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.49399999999999999}]},{"osd":6,"last update":"Tue Apr 23 03:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Apr 23 03:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.626},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.63200000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5684,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739468,"statfs":{"total":95995035648,"available":95989215232,"internally_reserved":0,"allocated":569344,"data_stored":237009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76300000000000001,"5min":0.996,"15min":0.996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.748,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78500000000000003,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56100000000000005}]},{"osd":2,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68400000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.67500000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.505},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.41799999999999998},"min":{"1min":0.32900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.43099999999999999,"15min":0.42399999999999999},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.54300000000000004}]},{"osd":6,"last update":"Tue Apr 23 03:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Apr 23 03:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T03:26:09.620 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-23T03:26:09.845 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:26:10.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:10 smithi007 ceph-mon[26683]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:26:10.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:10 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:10.515 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:26:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:10] "GET /metrics HTTP/1.1" 200 191460 "" "Prometheus/2.33.4" 2024-04-23T03:26:10.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:10 smithi007 ceph-mon[21886]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:26:10.516 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:10 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:10 smithi149 ceph-mon[22716]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:26:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:10 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:11.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:11 smithi007 ceph-mon[21886]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:11.277 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:11 smithi007 ceph-mon[26683]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:11.467 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:26:11.468 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-23T03:26:11.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:11 smithi149 ceph-mon[22716]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:12.132 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":38,"stamp":"2024-04-23T03:26:10.018993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48820,"kb_used_data":6068,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912396,"statfs":{"total":767960285184,"available":767910293504,"internally_reserved":0,"allocated":6213632,"data_stored":3487074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002471"},"pg_stats":[{"pgid":"1.0","version":"59'128","reported_seq":266,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-23T03:25:00.092995+0000","last_change":"2024-04-23T03:24:47.959672+0000","last_active":"2024-04-23T03:25:00.092995+0000","last_peered":"2024-04-23T03:25:00.092995+0000","last_clean":"2024-04-23T03:25:00.092995+0000","last_became_active":"2024-04-23T03:24:47.959296+0000","last_became_peered":"2024-04-23T03:24:47.959296+0000","last_unstale":"2024-04-23T03:25:00.092995+0000","last_undegraded":"2024-04-23T03:25:00.092995+0000","last_fullsized":"2024-04-23T03:25:00.092995+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-23T03:22:48.947740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-23T03:22:48.947740+0000","last_clean_scrub_stamp":"2024-04-23T03:22:48.947740+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-24T11:43:59.817136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234002,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6084,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739068,"statfs":{"total":95995035648,"available":95988805632,"internally_reserved":0,"allocated":1110016,"data_stored":766446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.495}]},{"osd":2,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.58399999999999996}]},{"osd":3,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.39500000000000002}]},{"osd":5,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.56799999999999995}]},{"osd":6,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55200000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":766769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.748}]},{"osd":2,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71599999999999997}]},{"osd":4,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.624}]},{"osd":5,"last update":"Tue Apr 23 03:25:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.55000000000000004}]},{"osd":7,"last update":"Tue Apr 23 03:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6328,"kb_used_data":560,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738824,"statfs":{"total":95995035648,"available":95988555776,"internally_reserved":0,"allocated":573440,"data_stored":237332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.35899999999999999,"15min":0.34300000000000003},"min":{"1min":0.27300000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.439,"5min":0.38700000000000001,"15min":0.36299999999999999},"min":{"1min":0.30299999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.65000000000000002,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46100000000000002}]},{"osd":2,"last update":"Tue Apr 23 03:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40699999999999997,"15min":0.39700000000000002},"min":{"1min":0.29799999999999999,"5min":0.252,"15min":0.252},"max":{"1min":0.58299999999999996,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.42199999999999999,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.628,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.40400000000000003}]},{"osd":3,"last update":"Tue Apr 23 03:25:19 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.47599999999999998,"15min":0.47699999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.47299999999999998,"15min":0.46800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41499999999999998}]},{"osd":4,"last update":"Tue Apr 23 03:25:35 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.629},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.628,"15min":0.626},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Apr 23 03:26:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.63200000000000001,"15min":0.629},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.60799999999999998},{"interface":"front","average":{"1min":1.0800000000000001,"5min":0.71799999999999997,"15min":0.65800000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":10.391,"5min":10.391,"15min":10.391},"last":0.70299999999999996}]},{"osd":6,"last update":"Tue Apr 23 03:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.64500000000000002}]},{"osd":7,"last update":"Tue Apr 23 03:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.67800000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6916,"kb_used_data":1084,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738236,"statfs":{"total":95995035648,"available":95987953664,"internally_reserved":0,"allocated":1110016,"data_stored":766504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Apr 23 03:25:31 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.374,"15min":0.35399999999999998},"min":{"1min":0.23599999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.40699999999999997,"15min":0.38100000000000001},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35199999999999998}]},{"osd":2,"last update":"Tue Apr 23 03:25:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41599999999999998,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.437,"15min":0.42699999999999999},"min":{"1min":0.28799999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.36199999999999999}]},{"osd":3,"last update":"Tue Apr 23 03:25:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.45200000000000001,"15min":0.45900000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.58999999999999997,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.622,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.33300000000000002}]},{"osd":4,"last update":"Tue Apr 23 03:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60799999999999998,"15min":0.60999999999999999},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83299999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.80600000000000005,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Apr 23 03:25:58 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.47399999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.61099999999999999,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.84299999999999997,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.749}]},{"osd":6,"last update":"Tue Apr 23 03:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.78600000000000003}]},{"osd":7,"last update":"Tue Apr 23 03:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.55300000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6272,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738880,"statfs":{"total":95995035648,"available":95988613120,"internally_reserved":0,"allocated":581632,"data_stored":237841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39000000000000001,"15min":0.376},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.29199999999999998}]},{"osd":1,"last update":"Tue Apr 23 03:25:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.40899999999999997,"15min":0.39600000000000002},"min":{"1min":0.30599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.42599999999999999,"15min":0.41199999999999998},"min":{"1min":0.29299999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.307}]},{"osd":3,"last update":"Tue Apr 23 03:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.35899999999999999}]},{"osd":4,"last update":"Tue Apr 23 03:25:42 2024","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.61199999999999999,"15min":0.60499999999999998},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.89100000000000001,"5min":1.0660000000000001,"15min":1.0660000000000001},"last":0.502}]},{"osd":5,"last update":"Tue Apr 23 03:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.83699999999999997,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.63400000000000001,"15min":0.64000000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.81399999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.42599999999999999}]},{"osd":6,"last update":"Tue Apr 23 03:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48399999999999999}]},{"osd":7,"last update":"Tue Apr 23 03:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.54900000000000004}]}]},{"osd":3,"up_from":25,"seq":107374182436,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.42899999999999999,"15min":0.42799999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.63400000000000001,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.433,"15min":0.435},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.47099999999999997}]},{"osd":1,"last update":"Tue Apr 23 03:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.44400000000000001,"15min":0.44700000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.57299999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.439,"15min":0.438},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.50800000000000001}]},{"osd":2,"last update":"Tue Apr 23 03:25:16 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46899999999999997,"15min":0.46999999999999997},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.59899999999999998,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.46800000000000003,"15min":0.46200000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70599999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.97599999999999998}]},{"osd":4,"last update":"Tue Apr 23 03:25:37 2024","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.61799999999999999,"15min":0.61499999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.80100000000000005}]},{"osd":5,"last update":"Tue Apr 23 03:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.625,"15min":0.626},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.89900000000000002}]},{"osd":6,"last update":"Tue Apr 23 03:25:22 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.81499999999999995}]},{"osd":7,"last update":"Tue Apr 23 03:25:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.91900000000000004}]}]},{"osd":4,"up_from":30,"seq":128849018911,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":237597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.58999999999999997,"15min":0.60199999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.70299999999999996,"5min":0.997,"15min":0.997},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.74099999999999999,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.68799999999999994}]},{"osd":1,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.75900000000000001,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.60299999999999998,"15min":0.61899999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.66800000000000004,"5min":1.105,"15min":1.105},"last":0.56200000000000006}]},{"osd":2,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.57999999999999996,"15min":0.59299999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71699999999999997,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.57699999999999996,"15min":0.58299999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.69099999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58599999999999997}]},{"osd":3,"last update":"Tue Apr 23 03:25:39 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.60899999999999999,"15min":0.61699999999999999},"min":{"1min":0.39700000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78000000000000003,"5min":0.874,"15min":0.874},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.57199999999999995,"15min":0.57999999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.69599999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.55400000000000005}]},{"osd":5,"last update":"Tue Apr 23 03:25:59 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.32400000000000001,"5min":0.315,"15min":0.315},"max":{"1min":0.52800000000000002,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.40799999999999997,"15min":0.40100000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.49399999999999999}]},{"osd":6,"last update":"Tue Apr 23 03:25:21 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Apr 23 03:25:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.626},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.63200000000000001}]}]},{"osd":5,"up_from":38,"seq":163208757275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5684,"kb_used_data":556,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739468,"statfs":{"total":95995035648,"available":95989215232,"internally_reserved":0,"allocated":569344,"data_stored":237009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76300000000000001,"5min":0.996,"15min":0.996},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56699999999999995,"15min":0.57199999999999995},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.748,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.45600000000000002}]},{"osd":1,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58599999999999997,"15min":0.58699999999999997},"min":{"1min":0.42599999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.78500000000000003,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.56399999999999995,"15min":0.56999999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70799999999999996,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56100000000000005}]},{"osd":2,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.55600000000000005,"15min":0.56000000000000005},"min":{"1min":0.41899999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.69699999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.68400000000000005,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56000000000000005,"15min":0.56599999999999995},"min":{"1min":0.40400000000000003,"5min":0.371,"15min":0.371},"max":{"1min":0.67500000000000004,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.505},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.61699999999999999,"15min":0.622},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.74099999999999999,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Apr 23 03:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.42499999999999999,"15min":0.41799999999999998},"min":{"1min":0.32900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.43099999999999999,"15min":0.42399999999999999},"min":{"1min":0.35699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.54300000000000004}]},{"osd":6,"last update":"Tue Apr 23 03:25:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Apr 23 03:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.63}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-23T03:26:12.133 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-23T03:26:12.133 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-23T03:26:12.134 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-23T03:26:12.134 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-23T03:26:12.358 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/mon.a/config 2024-04-23T03:26:12.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:12 smithi007 ceph-mon[26683]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:26:12.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:12 smithi007 ceph-mon[21886]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:26:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:12 smithi149 ceph-mon[22716]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-23T03:26:13.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:13 smithi007 ceph-mon[26683]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:13.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:13 smithi007 ceph-mon[21886]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:13 smithi149 ceph-mon[22716]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:14.235 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-23T03:26:14.236 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-23T03:26:14.786 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-23T03:26:14.787 INFO:tasks.cephadm:Setup complete, yielding 2024-04-23T03:26:14.787 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T03:26:14.798 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-23T03:26:14.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-23T03:26:15.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:14 smithi007 ceph-mon[26683]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:15.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:14 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/4256763564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T03:26:15.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:14 smithi007 ceph-mon[21886]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:15.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:14 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/4256763564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T03:26:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:14 smithi149 ceph-mon[22716]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:14 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4256763564' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-23T03:26:17.391 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T03:26:17.401 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-23T03:26:17.401 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-23T03:26:17.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:26:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:26:17.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:17 smithi007 ceph-mon[26683]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:17.978 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:17 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' 2024-04-23T03:26:17.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:17 smithi007 ceph-mon[21886]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:17.978 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:17 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' 2024-04-23T03:26:18.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:17 smithi149 ceph-mon[22716]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:18.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:17 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' 2024-04-23T03:26:19.123 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 podman[38828]: 2024-04-23 03:26:19.121847692 +0000 UTC m=+0.146728626 container create 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, description=Ceph Grafana Container, release=236.1648460182, distribution-scope=public, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-23T03:26:19.381 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 podman[38828]: 2024-04-23 03:26:19.348990214 +0000 UTC m=+0.373871106 container init 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, build-date=2022-03-28T10:36:18.413762, version=8.5, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , io.openshift.expose-services=, vcs-type=git) 2024-04-23T03:26:19.381 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 podman[38828]: 2024-04-23 03:26:19.351462655 +0000 UTC m=+0.376343547 container start 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, io.buildah.version=1.24.2, io.openshift.expose-services=, vcs-type=git, version=8.5, distribution-scope=public, description=Ceph Grafana Container, io.openshift.tags=base rhel8, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-04-23T03:26:19.631 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-23T03:26:19.632 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="App mode production" logger=settings 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-23T03:26:19.633 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-23T03:26:19.635 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-23T03:26:19.636 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-23T03:26:19.637 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-23T03:26:19.638 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-23T03:26:19.639 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-23T03:26:19.640 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-23T03:26:19.641 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 bash[38828]: 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 systemd[1]: Started Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-23T03:26:19.642 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-23T03:26:19.643 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-23T03:26:19.644 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-23T03:26:19.645 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-23T03:26:19.646 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-23T03:26:19.647 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-23T03:26:19.647 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-23T03:26:19.647 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-23T03:26:19.647 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-23T03:26:19.648 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-23T03:26:19.649 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-23T03:26:19.650 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-23T03:26:19.651 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-23T03:26:19.652 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[26683]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[21886]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:19 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:19.885 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-mon[22716]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:19.885 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:19.885 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:19.885 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:19.886 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-23T03:26:19.886 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-23T03:26:19.886 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-23T03:26:19.886 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-23T03:26:19.886 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-23T03:26:19.886 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-23T03:26:19.887 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-23T03:26:19.888 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-23T03:26:19.889 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-23T03:26:19.890 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-23T03:26:19.891 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-23T03:26:19.892 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-23T03:26:19.893 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-23T03:26:19.894 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-23T03:26:19.895 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-23T03:26:19.896 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=482.100538ms 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-23T03:26:19.897 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-23T03:26:20.172 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-23T03:26:20.172 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-23T03:26:20.173 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-23T03:26:20.173 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-23T03:26:20.173 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-23T03:26:20.173 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-23T03:26:20.173 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-23T03:26:20.173 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:26:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:26:19+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-23T03:26:20.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:20] "GET /metrics HTTP/1.1" 200 191464 "" "Prometheus/2.33.4" 2024-04-23T03:26:21.002 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:26:21.002 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "3ad28c19-b159-4c00-b550-9099f9b46dbb", 2024-04-23T03:26:21.002 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "r", 2024-04-23T03:26:21.002 INFO:teuthology.orchestra.run.smithi007.stdout: "current_period": "469f152c-d26c-4440-a7a3-8ae18916d24d", 2024-04-23T03:26:21.002 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1 2024-04-23T03:26:21.002 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:26:21.087 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:20 smithi149 ceph-mon[22716]: osdmap e60: 8 total, 8 up, 8 in 2024-04-23T03:26:21.087 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:20 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3552968233' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-23T03:26:21.087 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:20 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-23T03:26:21.087 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:20 smithi149 ceph-mon[22716]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:21.088 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:20 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[21886]: osdmap e60: 8 total, 8 up, 8 in 2024-04-23T03:26:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3552968233' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-23T03:26:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[21886]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[26683]: osdmap e60: 8 total, 8 up, 8 in 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3552968233' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[26683]: pgmap v44: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:26:21.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:20 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:21.758 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-23T03:26:22.048 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:21 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-23T03:26:22.048 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:21 smithi149 ceph-mon[22716]: osdmap e61: 8 total, 8 up, 8 in 2024-04-23T03:26:22.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:21 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-23T03:26:22.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:21 smithi007 ceph-mon[21886]: osdmap e61: 8 total, 8 up, 8 in 2024-04-23T03:26:22.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:21 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-23T03:26:22.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:21 smithi007 ceph-mon[26683]: osdmap e61: 8 total, 8 up, 8 in 2024-04-23T03:26:23.218 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:22 smithi149 ceph-mon[22716]: osdmap e62: 8 total, 8 up, 8 in 2024-04-23T03:26:23.219 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:22 smithi149 ceph-mon[22716]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-23T03:26:23.219 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:22 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:23.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:22 smithi007 ceph-mon[26683]: osdmap e62: 8 total, 8 up, 8 in 2024-04-23T03:26:23.313 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:22 smithi007 ceph-mon[26683]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-23T03:26:23.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:22 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:23.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:22 smithi007 ceph-mon[21886]: osdmap e62: 8 total, 8 up, 8 in 2024-04-23T03:26:23.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:22 smithi007 ceph-mon[21886]: pgmap v47: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-23T03:26:23.314 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:22 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:24.227 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:26:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "d47f1543-e101-4a9f-8138-429c9b2636a3", 2024-04-23T03:26:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2024-04-23T03:26:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "default", 2024-04-23T03:26:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": "true", 2024-04-23T03:26:24.227 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "", 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [], 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [], 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "", 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "3ad28c19-b159-4c00-b550-9099f9b46dbb", 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-04-23T03:26:24.228 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:24.229 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:26:24.893 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:24 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:24.893 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:24 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:24.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:24 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:24.893 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:24 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:24 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:24 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:25.708 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-23T03:26:26.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:25 smithi007 ceph-mon[26683]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-23T03:26:26.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:25 smithi007 ceph-mon[26683]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-23T03:26:26.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:25 smithi007 ceph-mon[26683]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-23T03:26:26.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:25 smithi007 ceph-mon[21886]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-23T03:26:26.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:25 smithi007 ceph-mon[21886]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-23T03:26:26.047 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:25 smithi007 ceph-mon[21886]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-23T03:26:26.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:25 smithi149 ceph-mon[22716]: pgmap v48: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-23T03:26:26.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:25 smithi149 ceph-mon[22716]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-23T03:26:26.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:25 smithi149 ceph-mon[22716]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-23T03:26:27.557 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:27 smithi007 bash[46274]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-23T03:26:27.747 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:26:27 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:26:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:27 smithi007 ceph-mon[21886]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-23T03:26:27.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:27 smithi007 ceph-mon[26683]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-23T03:26:28.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:27 smithi149 ceph-mon[22716]: pgmap v49: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 338 B/s rd, 1.5 KiB/s wr, 2 op/s 2024-04-23T03:26:28.457 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:28 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[44766]: level=info ts=2024-04-23T03:26:28.171Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:26:28.457 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:28 smithi007 podman[46298]: 2024-04-23 03:26:28.192268953 +0000 UTC m=+0.737537871 container died 2ea44ee13029fc958dadad4670c894c716d4be9c1e3f54e9d9e530a8662b8852 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:28.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:28 smithi007 podman[46298]: 2024-04-23 03:26:28.604723191 +0000 UTC m=+1.149992106 container cleanup 2ea44ee13029fc958dadad4670c894c716d4be9c1e3f54e9d9e530a8662b8852 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:28.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:28 smithi007 bash[46298]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a 2024-04-23T03:26:28.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:28 smithi007 bash[46350]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-23T03:26:29.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:28 smithi007 podman[46336]: 2024-04-23 03:26:28.931205262 +0000 UTC m=+0.734948027 container remove 2ea44ee13029fc958dadad4670c894c716d4be9c1e3f54e9d9e530a8662b8852 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:29.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:29 smithi007 podman[46336]: 2024-04-23 03:26:29.015718616 +0000 UTC m=+0.819461337 volume remove 7a232b96e096510c98976094ca175e4f029cd17da6369724490942eb64a929ff 2024-04-23T03:26:29.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:29 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-23T03:26:29.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:29 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:26:29.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:29 smithi007 systemd[1]: Starting Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:26:29.946 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:26:29.946 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "37144f88-9e74-47ce-b7a1-b0ed02a2e39f", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "z.rgw.meta:root", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "z.rgw.control", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "z.rgw.log", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "z.rgw.otp", 2024-04-23T03:26:29.947 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "", 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "" 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0 2024-04-23T03:26:29.948 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:29.949 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:29.949 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:26:29.949 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "3ad28c19-b159-4c00-b550-9099f9b46dbb", 2024-04-23T03:26:29.949 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-23T03:26:29.949 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:26:30.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:29 smithi007 ceph-mon[26683]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-23T03:26:30.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:29 smithi007 ceph-mon[21886]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-23T03:26:30.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:29 smithi149 ceph-mon[22716]: pgmap v50: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-23T03:26:30.559 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:26:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:30] "GET /metrics HTTP/1.1" 200 192962 "" "Prometheus/2.33.4" 2024-04-23T03:26:30.586 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-23T03:26:30.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:30 smithi007 podman[46541]: 2024-04-23 03:26:30.632345104 +0000 UTC m=+0.020154898 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-23T03:26:30.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:30 smithi007 podman[46541]: 2024-04-23 03:26:30.776440192 +0000 UTC m=+0.164249962 volume create ace53e1aad9216914ebd23fa5b3c58b1664e61d8064abf40c0af6fecdd7c3862 2024-04-23T03:26:31.130 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:30 smithi007 podman[46541]: 2024-04-23 03:26:30.868864561 +0000 UTC m=+0.256674372 container create 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:31.399 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 podman[46541]: 2024-04-23 03:26:31.398468371 +0000 UTC m=+0.786278161 container init 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:31.751 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 podman[46541]: 2024-04-23 03:26:31.401429298 +0000 UTC m=+0.789239074 container start 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:31.751 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.413Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-23T03:26:31.751 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.413Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-23T03:26:31.751 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.414Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-04-23T03:26:31.752 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.415Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-23T03:26:31.752 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.441Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-23T03:26:31.752 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.441Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-23T03:26:31.752 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.443Z caller=main.go:518 msg=Listening address=:9093 2024-04-23T03:26:31.752 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:31.443Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-23T03:26:32.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-mon[26683]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-04-23T03:26:32.067 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:31 smithi007 ceph-mon[21886]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-04-23T03:26:32.067 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 bash[46541]: 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d 2024-04-23T03:26:32.067 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:31 smithi007 systemd[1]: Started Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:26:32.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:31 smithi149 ceph-mon[22716]: pgmap v51: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 452 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-04-23T03:26:33.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:33.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[21886]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:26:33.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[21886]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:26:33.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[21886]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 815 B/s wr, 1 op/s 2024-04-23T03:26:33.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:33.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[26683]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:26:33.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[26683]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:26:33.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:32 smithi007 ceph-mon[26683]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 815 B/s wr, 1 op/s 2024-04-23T03:26:33.391 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:32 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:33.391 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:32 smithi149 ceph-mon[22716]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:26:33.391 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:32 smithi149 ceph-mon[22716]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:26:33.391 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:32 smithi149 ceph-mon[22716]: pgmap v52: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 713 B/s rd, 815 B/s wr, 1 op/s 2024-04-23T03:26:33.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:33.416Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00085406s 2024-04-23T03:26:35.019 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:34 smithi149 ceph-mon[22716]: osdmap e63: 8 total, 8 up, 8 in 2024-04-23T03:26:35.019 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:34 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3538943518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-23T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:34 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-23T03:26:35.020 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:34 smithi149 ceph-mon[22716]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 819 B/s wr, 1 op/s 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[26683]: osdmap e63: 8 total, 8 up, 8 in 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3538943518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[26683]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 819 B/s wr, 1 op/s 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[21886]: osdmap e63: 8 total, 8 up, 8 in 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3538943518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-23T03:26:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:34 smithi007 ceph-mon[21886]: pgmap v54: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 819 B/s wr, 1 op/s 2024-04-23T03:26:36.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:35 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-23T03:26:36.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:35 smithi149 ceph-mon[22716]: osdmap e64: 8 total, 8 up, 8 in 2024-04-23T03:26:36.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:35 smithi149 ceph-mon[22716]: osdmap e65: 8 total, 8 up, 8 in 2024-04-23T03:26:36.173 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 systemd[1]: Stopping Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:26:36.173 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 bash[39517]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-23T03:26:36.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:35 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-23T03:26:36.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:35 smithi007 ceph-mon[26683]: osdmap e64: 8 total, 8 up, 8 in 2024-04-23T03:26:36.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:35 smithi007 ceph-mon[26683]: osdmap e65: 8 total, 8 up, 8 in 2024-04-23T03:26:36.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:35 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-23T03:26:36.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:35 smithi007 ceph-mon[21886]: osdmap e64: 8 total, 8 up, 8 in 2024-04-23T03:26:36.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:35 smithi007 ceph-mon[21886]: osdmap e65: 8 total, 8 up, 8 in 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.189Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.190Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.190Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.190Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.190Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[37872]: ts=2024-04-23T03:26:36.190Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-23T03:26:36.475 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 podman[39530]: 2024-04-23 03:26:36.199642375 +0000 UTC m=+0.117063602 container died 167360a37db66824419f004b96933e2577527893fa9bc30cc9a183b7d0b42bc2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:36.807 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 podman[39530]: 2024-04-23 03:26:36.563905997 +0000 UTC m=+0.481327265 container cleanup 167360a37db66824419f004b96933e2577527893fa9bc30cc9a183b7d0b42bc2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:36.807 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 bash[39530]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a 2024-04-23T03:26:36.807 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 bash[39557]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-23T03:26:37.066 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3538943518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-23T03:26:37.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-23T03:26:37.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-mon[22716]: pgmap v57: 97 pgs: 28 creating+peering, 57 active+clean, 12 unknown; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T03:26:37.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-23T03:26:37.067 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:36 smithi149 ceph-mon[22716]: osdmap e66: 8 total, 8 up, 8 in 2024-04-23T03:26:37.067 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 podman[39544]: 2024-04-23 03:26:36.806580516 +0000 UTC m=+0.609273427 container remove 167360a37db66824419f004b96933e2577527893fa9bc30cc9a183b7d0b42bc2 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:37.067 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-23T03:26:37.067 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 systemd[1]: Stopped Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:26:37.067 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:36 smithi149 systemd[1]: Starting Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:26:37.067 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 podman[39622]: 2024-04-23 03:26:36.942733552 +0000 UTC m=+0.019455395 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-23T03:26:37.067 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 podman[39622]: 2024-04-23 03:26:37.065943136 +0000 UTC m=+0.142664951 container create fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3538943518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[21886]: pgmap v57: 97 pgs: 28 creating+peering, 57 active+clean, 12 unknown; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[21886]: osdmap e66: 8 total, 8 up, 8 in 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3538943518' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[26683]: pgmap v57: 97 pgs: 28 creating+peering, 57 active+clean, 12 unknown; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-23T03:26:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:36 smithi007 ceph-mon[26683]: osdmap e66: 8 total, 8 up, 8 in 2024-04-23T03:26:37.377 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 podman[39622]: 2024-04-23 03:26:37.284957703 +0000 UTC m=+0.361679521 container init fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:37.377 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 podman[39622]: 2024-04-23 03:26:37.287435585 +0000 UTC m=+0.364157401 container start fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:26:37.377 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.310Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-23T03:26:37.377 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.310Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-23T03:26:37.377 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.310Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-23T03:26:37.377 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.310Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi149 (none))" 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.310Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.310Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.312Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.313Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.313Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.315Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.315Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.511µs 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.315Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-23T03:26:37.378 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 bash[39622]: fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 2024-04-23T03:26:37.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 systemd[1]: Started Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:26:37.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.517Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-23T03:26:37.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.518Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-23T03:26:37.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.518Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.974µs wal_replay_duration=202.788675ms total_replay_duration=203.000147ms 2024-04-23T03:26:37.673 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.520Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-23T03:26:37.673 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.520Z caller=main.go:947 level=info msg="TSDB started" 2024-04-23T03:26:37.673 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.520Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-23T03:26:37.673 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.536Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.378289ms db_storage=925ns remote_storage=2.127µs web_handler=355ns query_engine=999ns scrape=328.337µs scrape_sd=23.601µs notify=21.786µs notify_sd=7.712µs rules=15.698067ms 2024-04-23T03:26:37.673 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:26:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:26:37.536Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-23T03:26:38.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-04-23T03:26:38.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-04-23T03:26:38.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:38.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:38.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: osdmap e67: 8 total, 8 up, 8 in 2024-04-23T03:26:38.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-23T03:26:38.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:38 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:26:38.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: osdmap e67: 8 total, 8 up, 8 in 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-23T03:26:38.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.149:9095"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:26:38.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.149:3000"}]: dispatch 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: osdmap e67: 8 total, 8 up, 8 in 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-23T03:26:38.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:38 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-23T03:26:39.770 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:39 smithi149 ceph-mon[22716]: pgmap v59: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-04-23T03:26:39.771 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:39 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-23T03:26:39.771 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:39 smithi149 ceph-mon[22716]: osdmap e68: 8 total, 8 up, 8 in 2024-04-23T03:26:39.771 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:39 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-23T03:26:39.771 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:39 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-23T03:26:39.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[26683]: pgmap v59: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-04-23T03:26:39.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[26683]: osdmap e68: 8 total, 8 up, 8 in 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[21886]: pgmap v59: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[21886]: osdmap e68: 8 total, 8 up, 8 in 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-23T03:26:39.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:39 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e06c083-37b2-45c0-8f36-7925dd43e4df", 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1, 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "469f152c-d26c-4440-a7a3-8ae18916d24d", 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "7e06c083-37b2-45c0-8f36-7925dd43e4df", 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-23T03:26:41.130 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "d47f1543-e101-4a9f-8138-429c9b2636a3", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "default", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": "true", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "37144f88-9e74-47ce-b7a1-b0ed02a2e39f", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "37144f88-9e74-47ce-b7a1-b0ed02a2e39f", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": "false", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": "false", 2024-04-23T03:26:41.131 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": "false", 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": "true", 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "" 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-23T03:26:41.132 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "3ad28c19-b159-4c00-b550-9099f9b46dbb", 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "37144f88-9e74-47ce-b7a1-b0ed02a2e39f", 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 597382703 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:41.133 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "d47f1543-e101-4a9f-8138-429c9b2636a3", 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "37144f88-9e74-47ce-b7a1-b0ed02a2e39f", 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-04-23T03:26:41.134 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-04-23T03:26:41.135 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "3ad28c19-b159-4c00-b550-9099f9b46dbb", 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "r", 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-04-23T03:26:41.136 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:26:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-04-23T03:26:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-23T03:26:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: osdmap e69: 8 total, 8 up, 8 in 2024-04-23T03:26:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-23T03:26:41.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-23T03:26:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:41.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:41 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: osdmap e69: 8 total, 8 up, 8 in 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: pgmap v62: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.6 KiB/s rd, 5 op/s 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: osdmap e69: 8 total, 8 up, 8 in 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3969826342' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:41.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:41.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:26:41 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:26:41.418Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002292319s 2024-04-23T03:26:41.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-23T03:26:42.201 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:42 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-23T03:26:42.201 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:42 smithi007 ceph-mon[21886]: osdmap e70: 8 total, 8 up, 8 in 2024-04-23T03:26:42.202 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:42 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-23T03:26:42.202 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:42 smithi007 ceph-mon[26683]: osdmap e70: 8 total, 8 up, 8 in 2024-04-23T03:26:42.330 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:42 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-23T03:26:42.330 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:42 smithi149 ceph-mon[22716]: osdmap e70: 8 total, 8 up, 8 in 2024-04-23T03:26:43.282 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:43 smithi007 ceph-mon[26683]: pgmap v65: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-23T03:26:43.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:43 smithi007 ceph-mon[21886]: pgmap v65: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-23T03:26:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:43 smithi149 ceph-mon[22716]: pgmap v65: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-23T03:26:43.930 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled rgw.foo update... 2024-04-23T03:26:44.550 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch apply rgw smpl' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: Saving service rgw.foo spec with placement count:2 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.072 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: Saving service rgw.foo spec with placement count:2 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.073 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: Saving service rgw.foo spec with placement count:2 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:45.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:46.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:45 smithi007 ceph-mon[26683]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 171 B/s rd, 343 B/s wr, 1 op/s 2024-04-23T03:26:46.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:45 smithi007 ceph-mon[26683]: Saving service rgw.foo spec with placement count:2 2024-04-23T03:26:46.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:45 smithi007 ceph-mon[26683]: Deploying daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:26:46.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:45 smithi007 ceph-mon[21886]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 171 B/s rd, 343 B/s wr, 1 op/s 2024-04-23T03:26:46.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:45 smithi007 ceph-mon[21886]: Saving service rgw.foo spec with placement count:2 2024-04-23T03:26:46.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:45 smithi007 ceph-mon[21886]: Deploying daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:26:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:45 smithi149 ceph-mon[22716]: pgmap v66: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 171 B/s rd, 343 B/s wr, 1 op/s 2024-04-23T03:26:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:45 smithi149 ceph-mon[22716]: Saving service rgw.foo spec with placement count:2 2024-04-23T03:26:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:45 smithi149 ceph-mon[22716]: Deploying daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:26:47.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:46 smithi007 ceph-mon[26683]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-23T03:26:47.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:46 smithi007 ceph-mon[21886]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-23T03:26:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:46 smithi149 ceph-mon[22716]: pgmap v67: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-04-23T03:26:47.652 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled rgw.smpl update... 2024-04-23T03:26:47.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:26:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:26:48.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph osd pool create foo' 2024-04-23T03:26:48.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:48 smithi007 ceph-mon[26683]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:26:48.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:48 smithi007 ceph-mon[26683]: Saving service rgw.smpl spec with placement count:2 2024-04-23T03:26:48.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:48 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:48.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:48 smithi007 ceph-mon[21886]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:26:48.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:48 smithi007 ceph-mon[21886]: Saving service rgw.smpl spec with placement count:2 2024-04-23T03:26:48.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:48 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:48.908 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:48 smithi149 ceph-mon[22716]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:26:48.908 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:48 smithi149 ceph-mon[22716]: Saving service rgw.smpl spec with placement count:2 2024-04-23T03:26:48.908 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:48 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-23T03:26:50.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[26683]: Deploying daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:50.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:50.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:49 smithi007 ceph-mon[21886]: Deploying daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:26:50.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-23T03:26:50.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:50.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:50.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:50.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:50.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:49 smithi149 ceph-mon[22716]: Deploying daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:26:50.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:26:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:50] "GET /metrics HTTP/1.1" 200 196711 "" "Prometheus/2.33.4" 2024-04-23T03:26:51.038 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'foo' created 2024-04-23T03:26:51.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:51 smithi007 ceph-mon[26683]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 822 B/s rd, 924 B/s wr, 2 op/s 2024-04-23T03:26:51.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:51 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1697015302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-23T03:26:51.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:51 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-23T03:26:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:51 smithi007 ceph-mon[21886]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 822 B/s rd, 924 B/s wr, 2 op/s 2024-04-23T03:26:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:51 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1697015302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-23T03:26:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:51 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-23T03:26:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:51 smithi149 ceph-mon[22716]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 822 B/s rd, 924 B/s wr, 2 op/s 2024-04-23T03:26:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:51 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1697015302' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-23T03:26:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:51 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-23T03:26:51.862 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'rbd pool init foo' 2024-04-23T03:26:52.238 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[26683]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-23T03:26:52.238 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[26683]: osdmap e71: 8 total, 8 up, 8 in 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[26683]: osdmap e72: 8 total, 8 up, 8 in 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[21886]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[21886]: osdmap e71: 8 total, 8 up, 8 in 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:52.239 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:52 smithi007 ceph-mon[21886]: osdmap e72: 8 total, 8 up, 8 in 2024-04-23T03:26:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:52 smithi149 ceph-mon[22716]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-23T03:26:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:52 smithi149 ceph-mon[22716]: osdmap e71: 8 total, 8 up, 8 in 2024-04-23T03:26:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:52 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:52.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:52 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:52.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:52 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:26:52.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:52 smithi149 ceph-mon[22716]: osdmap e72: 8 total, 8 up, 8 in 2024-04-23T03:26:53.101 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:53 smithi007 ceph-mon[26683]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 5.1 KiB/s wr, 209 op/s 2024-04-23T03:26:53.101 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:53 smithi007 ceph-mon[21886]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 5.1 KiB/s wr, 209 op/s 2024-04-23T03:26:53.374 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:53 smithi149 ceph-mon[22716]: pgmap v72: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 5.1 KiB/s wr, 209 op/s 2024-04-23T03:26:54.160 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:54 smithi007 ceph-mon[26683]: osdmap e73: 8 total, 8 up, 8 in 2024-04-23T03:26:54.160 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:54 smithi007 ceph-mon[21886]: osdmap e73: 8 total, 8 up, 8 in 2024-04-23T03:26:54.353 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:54 smithi149 ceph-mon[22716]: osdmap e73: 8 total, 8 up, 8 in 2024-04-23T03:26:55.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:55 smithi007 ceph-mon[26683]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 5.7 KiB/s wr, 276 op/s 2024-04-23T03:26:55.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:55 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/214145141' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-23T03:26:55.137 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:55 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:55.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:55 smithi007 ceph-mon[21886]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 5.7 KiB/s wr, 276 op/s 2024-04-23T03:26:55.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:55 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/214145141' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-23T03:26:55.138 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:55 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:55 smithi149 ceph-mon[22716]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 132 KiB/s rd, 5.7 KiB/s wr, 276 op/s 2024-04-23T03:26:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:55 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/214145141' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-23T03:26:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:55 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:56.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:56 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/214145141' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-23T03:26:56.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:56 smithi007 ceph-mon[26683]: osdmap e74: 8 total, 8 up, 8 in 2024-04-23T03:26:56.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:56 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:56 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/214145141' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-23T03:26:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:56 smithi007 ceph-mon[21886]: osdmap e74: 8 total, 8 up, 8 in 2024-04-23T03:26:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:56 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:56 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/214145141' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-23T03:26:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:56 smithi149 ceph-mon[22716]: osdmap e74: 8 total, 8 up, 8 in 2024-04-23T03:26:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:56 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 253 KiB/s rd, 3.4 KiB/s wr, 435 op/s 2024-04-23T03:26:57.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: osdmap e75: 8 total, 8 up, 8 in 2024-04-23T03:26:57.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: Checking dashboard <-> RGW credentials 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: Saving service rgw.smpl spec with placement count:2 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:57.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[26683]: Deploying daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 253 KiB/s rd, 3.4 KiB/s wr, 435 op/s 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: osdmap e75: 8 total, 8 up, 8 in 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: Checking dashboard <-> RGW credentials 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: Saving service rgw.smpl spec with placement count:2 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:57.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:57 smithi007 ceph-mon[21886]: Deploying daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:26:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 253 KiB/s rd, 3.4 KiB/s wr, 435 op/s 2024-04-23T03:26:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: osdmap e75: 8 total, 8 up, 8 in 2024-04-23T03:26:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: Checking dashboard <-> RGW credentials 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: Saving service rgw.smpl spec with placement count:2 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:26:57.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-mon[22716]: Deploying daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:26:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:26:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:26:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:58 smithi007 ceph-mon[21886]: osdmap e76: 8 total, 8 up, 8 in 2024-04-23T03:26:58.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:58 smithi007 ceph-mon[26683]: osdmap e76: 8 total, 8 up, 8 in 2024-04-23T03:26:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:58 smithi149 ceph-mon[22716]: osdmap e76: 8 total, 8 up, 8 in 2024-04-23T03:26:58.688 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-23T03:26:59.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:26:59 smithi007 ceph-mon[26683]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 0 B/s wr, 273 op/s 2024-04-23T03:26:59.338 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:26:59 smithi007 ceph-mon[21886]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 0 B/s wr, 273 op/s 2024-04-23T03:26:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:26:59 smithi149 ceph-mon[22716]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 177 KiB/s rd, 0 B/s wr, 273 op/s 2024-04-23T03:27:00.667 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:00] "GET /metrics HTTP/1.1" 200 210523 "" "Prometheus/2.33.4" 2024-04-23T03:27:00.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:00.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:27:00.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: Deploying daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:00.668 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: Deploying daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:27:00.669 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:27:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:27:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-23T03:27:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: Deploying daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]: dispatch 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:27:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:27:01.096 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled iscsi.foo update... 2024-04-23T03:27:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 0 B/s wr, 227 op/s 2024-04-23T03:27:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-04-23T03:27:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-23T03:27:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-23T03:27:01.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-23T03:27:01.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: osdmap e77: 8 total, 8 up, 8 in 2024-04-23T03:27:01.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:01 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:01.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 0 B/s wr, 227 op/s 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: osdmap e77: 8 total, 8 up, 8 in 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 64 MiB used, 715 GiB / 715 GiB avail; 147 KiB/s rd, 0 B/s wr, 227 op/s 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [7, 2]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: osdmap e77: 8 total, 8 up, 8 in 2024-04-23T03:27:01.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:01 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:01.846 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'sleep 120' 2024-04-23T03:27:02.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[26683]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:27:02.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[26683]: Saving service iscsi.foo spec with placement count:1 2024-04-23T03:27:02.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[26683]: osdmap e78: 8 total, 8 up, 8 in 2024-04-23T03:27:02.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:02.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:02.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:02.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[21886]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:27:02.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[21886]: Saving service iscsi.foo spec with placement count:1 2024-04-23T03:27:02.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[21886]: osdmap e78: 8 total, 8 up, 8 in 2024-04-23T03:27:02.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:02.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:02.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:02 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:02.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:02 smithi149 ceph-mon[22716]: from='client.14817 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:27:02.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:02 smithi149 ceph-mon[22716]: Saving service iscsi.foo spec with placement count:1 2024-04-23T03:27:02.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:02 smithi149 ceph-mon[22716]: osdmap e78: 8 total, 8 up, 8 in 2024-04-23T03:27:02.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:02 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:02.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:02 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:02.839 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:02 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:03.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:03 smithi007 ceph-mon[21886]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 856 B/s wr, 287 op/s 2024-04-23T03:27:03.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:03 smithi007 ceph-mon[26683]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 856 B/s wr, 287 op/s 2024-04-23T03:27:03.834 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:03.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:27:03.834 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:03.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:27:03.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:03 smithi149 ceph-mon[22716]: pgmap v83: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 190 KiB/s rd, 856 B/s wr, 287 op/s 2024-04-23T03:27:05.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:04 smithi149 ceph-mon[22716]: osdmap e79: 8 total, 8 up, 8 in 2024-04-23T03:27:05.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:04 smithi149 ceph-mon[22716]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 189 KiB/s rd, 853 B/s wr, 286 op/s 2024-04-23T03:27:05.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:04 smithi007 ceph-mon[21886]: osdmap e79: 8 total, 8 up, 8 in 2024-04-23T03:27:05.220 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:04 smithi007 ceph-mon[21886]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 189 KiB/s rd, 853 B/s wr, 286 op/s 2024-04-23T03:27:05.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:04 smithi007 ceph-mon[26683]: osdmap e79: 8 total, 8 up, 8 in 2024-04-23T03:27:05.220 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:04 smithi007 ceph-mon[26683]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 189 KiB/s rd, 853 B/s wr, 286 op/s 2024-04-23T03:27:06.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:06 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:06.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:06 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:06.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:06 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:06.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:06 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:06.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:06 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:06.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:06 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 351 KiB/s rd, 1023 B/s wr, 534 op/s; 7 B/s, 0 objects/s recovering 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: Checking dashboard <-> RGW credentials 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: Checking pool "foo" exists for service iscsi.foo 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[26683]: Deploying daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:27:07.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 351 KiB/s rd, 1023 B/s wr, 534 op/s; 7 B/s, 0 objects/s recovering 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: Checking dashboard <-> RGW credentials 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: Checking pool "foo" exists for service iscsi.foo 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:07.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:07 smithi007 ceph-mon[21886]: Deploying daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 351 KiB/s rd, 1023 B/s wr, 534 op/s; 7 B/s, 0 objects/s recovering 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: Checking dashboard <-> RGW credentials 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: Checking pool "foo" exists for service iscsi.foo 2024-04-23T03:27:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:27:07.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-23T03:27:07.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:07.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:07 smithi149 ceph-mon[22716]: Deploying daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:27:08.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:08 smithi007 ceph-mon[26683]: mgrmap e21: y(active, since 2m), standbys: x 2024-04-23T03:27:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:08 smithi007 ceph-mon[21886]: mgrmap e21: y(active, since 2m), standbys: x 2024-04-23T03:27:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:08 smithi149 ceph-mon[22716]: mgrmap e21: y(active, since 2m), standbys: x 2024-04-23T03:27:09.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:09 smithi007 ceph-mon[26683]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 282 KiB/s rd, 821 B/s wr, 428 op/s; 6 B/s, 0 objects/s recovering 2024-04-23T03:27:09.720 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:09 smithi007 ceph-mon[21886]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 282 KiB/s rd, 821 B/s wr, 428 op/s; 6 B/s, 0 objects/s recovering 2024-04-23T03:27:09.823 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:09 smithi149 ceph-mon[22716]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 282 KiB/s rd, 821 B/s wr, 428 op/s; 6 B/s, 0 objects/s recovering 2024-04-23T03:27:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/169832967' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:27:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/164924966' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1711557554"}]: dispatch 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/169832967' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/164924966' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1711557554"}]: dispatch 2024-04-23T03:27:10.834 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:27:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:10] "GET /metrics HTTP/1.1" 200 210523 "" "Prometheus/2.33.4" 2024-04-23T03:27:10.926 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:10 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:10.927 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:10 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:10.927 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:10 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:10.927 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:10 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:10.927 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:10 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/169832967' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:27:10.927 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:10 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/164924966' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1711557554"}]: dispatch 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[21886]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 120 B/s wr, 175 op/s; 5 B/s, 0 objects/s recovering 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/164924966' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1711557554"}]': finished 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[21886]: osdmap e80: 8 total, 8 up, 8 in 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/650168001' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/136969602"}]: dispatch 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[26683]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 120 B/s wr, 175 op/s; 5 B/s, 0 objects/s recovering 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/164924966' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1711557554"}]': finished 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[26683]: osdmap e80: 8 total, 8 up, 8 in 2024-04-23T03:27:11.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:11 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/650168001' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/136969602"}]: dispatch 2024-04-23T03:27:11.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:11 smithi149 ceph-mon[22716]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 120 B/s wr, 175 op/s; 5 B/s, 0 objects/s recovering 2024-04-23T03:27:11.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:11 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/164924966' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1711557554"}]': finished 2024-04-23T03:27:11.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:11 smithi149 ceph-mon[22716]: osdmap e80: 8 total, 8 up, 8 in 2024-04-23T03:27:11.982 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:11 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/650168001' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/136969602"}]: dispatch 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 254 B/s wr, 213 op/s; 5 B/s, 0 objects/s recovering 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/650168001' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/136969602"}]': finished 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: osdmap e81: 8 total, 8 up, 8 in 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/287953017' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]: dispatch 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]: dispatch 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 254 B/s wr, 213 op/s; 5 B/s, 0 objects/s recovering 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/650168001' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/136969602"}]': finished 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: osdmap e81: 8 total, 8 up, 8 in 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/287953017' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-23T03:27:13.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:13.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 141 KiB/s rd, 254 B/s wr, 213 op/s; 5 B/s, 0 objects/s recovering 2024-04-23T03:27:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/650168001' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/136969602"}]': finished 2024-04-23T03:27:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: osdmap e81: 8 total, 8 up, 8 in 2024-04-23T03:27:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/287953017' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]: dispatch 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]: dispatch 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:27:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:27:13.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:27:13.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:27:13.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:27:13.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:13.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:27:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-23T03:27:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2024-04-23T03:27:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-23T03:27:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-23T03:27:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: Checking dashboard <-> RGW credentials 2024-04-23T03:27:14.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]': finished 2024-04-23T03:27:14.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: osdmap e82: 8 total, 8 up, 8 in 2024-04-23T03:27:14.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:14.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1695807225' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]: dispatch 2024-04-23T03:27:14.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:14 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]: dispatch 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: Checking dashboard <-> RGW credentials 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]': finished 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: osdmap e82: 8 total, 8 up, 8 in 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1695807225' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]: dispatch 2024-04-23T03:27:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]: dispatch 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: Checking dashboard <-> RGW credentials 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2530963392"}]': finished 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: osdmap e82: 8 total, 8 up, 8 in 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1695807225' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]: dispatch 2024-04-23T03:27:14.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:14 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]: dispatch 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[26683]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 170 B/s wr, 39 op/s 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]': finished 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[26683]: osdmap e83: 8 total, 8 up, 8 in 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3267058039' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3100710170"}]: dispatch 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[21886]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 170 B/s wr, 39 op/s 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]': finished 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[21886]: osdmap e83: 8 total, 8 up, 8 in 2024-04-23T03:27:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:15 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3267058039' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3100710170"}]: dispatch 2024-04-23T03:27:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:15 smithi149 ceph-mon[22716]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 170 B/s wr, 39 op/s 2024-04-23T03:27:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:15 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3100710170"}]': finished 2024-04-23T03:27:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:15 smithi149 ceph-mon[22716]: osdmap e83: 8 total, 8 up, 8 in 2024-04-23T03:27:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:15 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3267058039' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3100710170"}]: dispatch 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3267058039' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3100710170"}]': finished 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[26683]: osdmap e84: 8 total, 8 up, 8 in 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1314042069' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]: dispatch 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]: dispatch 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3267058039' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3100710170"}]': finished 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[21886]: osdmap e84: 8 total, 8 up, 8 in 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1314042069' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]: dispatch 2024-04-23T03:27:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:16 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]: dispatch 2024-04-23T03:27:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:16 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3267058039' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3100710170"}]': finished 2024-04-23T03:27:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:16 smithi149 ceph-mon[22716]: osdmap e84: 8 total, 8 up, 8 in 2024-04-23T03:27:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:16 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1314042069' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]: dispatch 2024-04-23T03:27:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:16 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]: dispatch 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[26683]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 25 op/s 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]': finished 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[26683]: osdmap e85: 8 total, 8 up, 8 in 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1000706462' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/752314468"}]: dispatch 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[21886]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 25 op/s 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]': finished 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[21886]: osdmap e85: 8 total, 8 up, 8 in 2024-04-23T03:27:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:17 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1000706462' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/752314468"}]: dispatch 2024-04-23T03:27:17.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:27:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:27:17.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:17 smithi149 ceph-mon[22716]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 25 op/s 2024-04-23T03:27:17.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:17 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1785735127"}]': finished 2024-04-23T03:27:17.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:17 smithi149 ceph-mon[22716]: osdmap e85: 8 total, 8 up, 8 in 2024-04-23T03:27:17.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:17 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1000706462' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/752314468"}]: dispatch 2024-04-23T03:27:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1000706462' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/752314468"}]': finished 2024-04-23T03:27:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[21886]: osdmap e86: 8 total, 8 up, 8 in 2024-04-23T03:27:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1491192497' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]: dispatch 2024-04-23T03:27:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]: dispatch 2024-04-23T03:27:18.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1000706462' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/752314468"}]': finished 2024-04-23T03:27:18.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[26683]: osdmap e86: 8 total, 8 up, 8 in 2024-04-23T03:27:18.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1491192497' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]: dispatch 2024-04-23T03:27:18.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:18 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]: dispatch 2024-04-23T03:27:18.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:18 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1000706462' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/752314468"}]': finished 2024-04-23T03:27:18.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:18 smithi149 ceph-mon[22716]: osdmap e86: 8 total, 8 up, 8 in 2024-04-23T03:27:18.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:18 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1491192497' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]: dispatch 2024-04-23T03:27:18.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:18 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]: dispatch 2024-04-23T03:27:19.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:19 smithi149 ceph-mon[22716]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 25 op/s 2024-04-23T03:27:19.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:19 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]': finished 2024-04-23T03:27:19.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:19 smithi149 ceph-mon[22716]: osdmap e87: 8 total, 8 up, 8 in 2024-04-23T03:27:19.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:19 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4152386771' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]: dispatch 2024-04-23T03:27:19.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:19 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]: dispatch 2024-04-23T03:27:19.807 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[26683]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 25 op/s 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]': finished 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[26683]: osdmap e87: 8 total, 8 up, 8 in 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/4152386771' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]: dispatch 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]: dispatch 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[21886]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 25 op/s 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/2136965699"}]': finished 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[21886]: osdmap e87: 8 total, 8 up, 8 in 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/4152386771' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]: dispatch 2024-04-23T03:27:19.808 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:19 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]: dispatch 2024-04-23T03:27:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:20 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]': finished 2024-04-23T03:27:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:20 smithi149 ceph-mon[22716]: osdmap e88: 8 total, 8 up, 8 in 2024-04-23T03:27:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:20 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3833447329' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]: dispatch 2024-04-23T03:27:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:20 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]: dispatch 2024-04-23T03:27:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:20 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:20] "GET /metrics HTTP/1.1" 200 214914 "" "Prometheus/2.33.4" 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]': finished 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[21886]: osdmap e88: 8 total, 8 up, 8 in 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3833447329' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]: dispatch 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]: dispatch 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/2136965699"}]': finished 2024-04-23T03:27:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[26683]: osdmap e88: 8 total, 8 up, 8 in 2024-04-23T03:27:20.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3833447329' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]: dispatch 2024-04-23T03:27:20.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]: dispatch 2024-04-23T03:27:20.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:20 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:21.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:21 smithi149 ceph-mon[22716]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:27:21.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:21 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]': finished 2024-04-23T03:27:21.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:21 smithi149 ceph-mon[22716]: osdmap e89: 8 total, 8 up, 8 in 2024-04-23T03:27:21.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:21 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1718405880' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3203492347"}]: dispatch 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[26683]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]': finished 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[26683]: osdmap e89: 8 total, 8 up, 8 in 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1718405880' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3203492347"}]: dispatch 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[21886]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3759347266"}]': finished 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[21886]: osdmap e89: 8 total, 8 up, 8 in 2024-04-23T03:27:21.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:21 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1718405880' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3203492347"}]: dispatch 2024-04-23T03:27:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:22 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1718405880' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3203492347"}]': finished 2024-04-23T03:27:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:22 smithi149 ceph-mon[22716]: osdmap e90: 8 total, 8 up, 8 in 2024-04-23T03:27:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:22 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3487760319' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2924706414"}]: dispatch 2024-04-23T03:27:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:22 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1718405880' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3203492347"}]': finished 2024-04-23T03:27:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:22 smithi007 ceph-mon[21886]: osdmap e90: 8 total, 8 up, 8 in 2024-04-23T03:27:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:22 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3487760319' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2924706414"}]: dispatch 2024-04-23T03:27:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:22 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1718405880' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3203492347"}]': finished 2024-04-23T03:27:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:22 smithi007 ceph-mon[26683]: osdmap e90: 8 total, 8 up, 8 in 2024-04-23T03:27:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:22 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3487760319' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2924706414"}]: dispatch 2024-04-23T03:27:23.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:23 smithi149 ceph-mon[22716]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:23.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:23 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3487760319' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2924706414"}]': finished 2024-04-23T03:27:23.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:23 smithi149 ceph-mon[22716]: osdmap e91: 8 total, 8 up, 8 in 2024-04-23T03:27:23.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:23 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4188996539' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]: dispatch 2024-04-23T03:27:23.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:23 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]: dispatch 2024-04-23T03:27:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[21886]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3487760319' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2924706414"}]': finished 2024-04-23T03:27:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[21886]: osdmap e91: 8 total, 8 up, 8 in 2024-04-23T03:27:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/4188996539' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]: dispatch 2024-04-23T03:27:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]: dispatch 2024-04-23T03:27:23.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[26683]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:23.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3487760319' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2924706414"}]': finished 2024-04-23T03:27:23.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[26683]: osdmap e91: 8 total, 8 up, 8 in 2024-04-23T03:27:23.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/4188996539' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]: dispatch 2024-04-23T03:27:23.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]: dispatch 2024-04-23T03:27:23.834 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:27:23.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:27:23.835 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:27:23.835 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:23.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:27:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:24 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]': finished 2024-04-23T03:27:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:24 smithi149 ceph-mon[22716]: osdmap e92: 8 total, 8 up, 8 in 2024-04-23T03:27:24.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:24 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2631184402' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]: dispatch 2024-04-23T03:27:24.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:24 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]: dispatch 2024-04-23T03:27:24.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]': finished 2024-04-23T03:27:24.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[21886]: osdmap e92: 8 total, 8 up, 8 in 2024-04-23T03:27:24.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2631184402' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]: dispatch 2024-04-23T03:27:24.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]: dispatch 2024-04-23T03:27:24.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3542800857"}]': finished 2024-04-23T03:27:24.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[26683]: osdmap e92: 8 total, 8 up, 8 in 2024-04-23T03:27:24.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2631184402' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]: dispatch 2024-04-23T03:27:24.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:24 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]: dispatch 2024-04-23T03:27:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:25 smithi149 ceph-mon[22716]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:25 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]': finished 2024-04-23T03:27:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:25 smithi149 ceph-mon[22716]: osdmap e93: 8 total, 8 up, 8 in 2024-04-23T03:27:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:25 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3120237533' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2507582309"}]: dispatch 2024-04-23T03:27:25.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[21886]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:25.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]': finished 2024-04-23T03:27:25.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[21886]: osdmap e93: 8 total, 8 up, 8 in 2024-04-23T03:27:25.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3120237533' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2507582309"}]: dispatch 2024-04-23T03:27:25.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[26683]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:25.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/860775767"}]': finished 2024-04-23T03:27:25.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[26683]: osdmap e93: 8 total, 8 up, 8 in 2024-04-23T03:27:25.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:25 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3120237533' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2507582309"}]: dispatch 2024-04-23T03:27:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:26 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3120237533' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2507582309"}]': finished 2024-04-23T03:27:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:26 smithi149 ceph-mon[22716]: osdmap e94: 8 total, 8 up, 8 in 2024-04-23T03:27:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:26 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1504514286' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3542800857"}]: dispatch 2024-04-23T03:27:26.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:26 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3120237533' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2507582309"}]': finished 2024-04-23T03:27:26.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:26 smithi007 ceph-mon[26683]: osdmap e94: 8 total, 8 up, 8 in 2024-04-23T03:27:26.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:26 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1504514286' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3542800857"}]: dispatch 2024-04-23T03:27:26.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:26 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3120237533' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2507582309"}]': finished 2024-04-23T03:27:26.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:26 smithi007 ceph-mon[21886]: osdmap e94: 8 total, 8 up, 8 in 2024-04-23T03:27:26.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:26 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1504514286' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3542800857"}]: dispatch 2024-04-23T03:27:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:27 smithi149 ceph-mon[22716]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:27 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1504514286' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3542800857"}]': finished 2024-04-23T03:27:27.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:27 smithi149 ceph-mon[22716]: osdmap e95: 8 total, 8 up, 8 in 2024-04-23T03:27:27.673 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:27:27 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:27:27.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:27 smithi007 ceph-mon[26683]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:27.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:27 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1504514286' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3542800857"}]': finished 2024-04-23T03:27:27.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:27 smithi007 ceph-mon[26683]: osdmap e95: 8 total, 8 up, 8 in 2024-04-23T03:27:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:27 smithi007 ceph-mon[21886]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:27 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1504514286' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3542800857"}]': finished 2024-04-23T03:27:27.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:27 smithi007 ceph-mon[21886]: osdmap e95: 8 total, 8 up, 8 in 2024-04-23T03:27:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:29 smithi149 ceph-mon[22716]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:27:29.818 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:29 smithi007 ceph-mon[26683]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:27:29.819 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:29 smithi007 ceph-mon[21886]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:27:30.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:30 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:30.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:30 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:30.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:30 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:30.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:27:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:30] "GET /metrics HTTP/1.1" 200 214906 "" "Prometheus/2.33.4" 2024-04-23T03:27:31.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:31 smithi149 ceph-mon[22716]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:31.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:31 smithi007 ceph-mon[21886]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:31.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:31 smithi007 ceph-mon[26683]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:33.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:33 smithi149 ceph-mon[22716]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:27:33.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:33 smithi007 ceph-mon[21886]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:27:33.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:33 smithi007 ceph-mon[26683]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:27:33.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:27:33.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:27:33.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:27:33.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:27:33.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:27:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:34 smithi007 ceph-mon[26683]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:34 smithi007 ceph-mon[21886]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:34 smithi149 ceph-mon[22716]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:37 smithi149 ceph-mon[22716]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:27:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:37 smithi007 ceph-mon[21886]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:27:37.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:37 smithi007 ceph-mon[26683]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:27:37.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:27:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:27:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:39 smithi149 ceph-mon[22716]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-04-23T03:27:39.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:39 smithi007 ceph-mon[26683]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-04-23T03:27:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:39 smithi007 ceph-mon[21886]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 0 op/s 2024-04-23T03:27:40.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:40 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:40.490 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:40 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:40.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:40 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:40.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:27:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:40] "GET /metrics HTTP/1.1" 200 214906 "" "Prometheus/2.33.4" 2024-04-23T03:27:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:41 smithi149 ceph-mon[22716]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:41 smithi007 ceph-mon[26683]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:41 smithi007 ceph-mon[21886]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:43 smithi149 ceph-mon[22716]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:43.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:43 smithi007 ceph-mon[26683]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:43.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:43 smithi007 ceph-mon[21886]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:43.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:27:43 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:27:43.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:27:45.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:44 smithi007 ceph-mon[26683]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:45.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:44 smithi007 ceph-mon[21886]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:44 smithi149 ceph-mon[22716]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:47 smithi149 ceph-mon[22716]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:47.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:47 smithi007 ceph-mon[21886]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:47 smithi007 ceph-mon[26683]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:47.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:27:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:27:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:49 smithi149 ceph-mon[22716]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:49.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:49 smithi007 ceph-mon[21886]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:49.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:49 smithi007 ceph-mon[26683]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:50 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:50.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:50 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:50.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:50 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:27:50.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:27:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:50] "GET /metrics HTTP/1.1" 200 214829 "" "Prometheus/2.33.4" 2024-04-23T03:27:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:51 smithi149 ceph-mon[22716]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:51.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:51 smithi007 ceph-mon[26683]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:51 smithi007 ceph-mon[21886]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:53.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:53 smithi149 ceph-mon[22716]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:53.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:53 smithi007 ceph-mon[26683]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:53 smithi007 ceph-mon[21886]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:54 smithi007 ceph-mon[26683]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:55.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:54 smithi007 ceph-mon[21886]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:54 smithi149 ceph-mon[22716]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:57 smithi149 ceph-mon[22716]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:57 smithi007 ceph-mon[26683]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:57 smithi007 ceph-mon[21886]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:27:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:27:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:27:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:27:59 smithi149 ceph-mon[22716]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:27:59 smithi007 ceph-mon[26683]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:27:59.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:27:59 smithi007 ceph-mon[21886]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:00 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:28:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:28:00.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:00.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:28:00.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:28:00.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:00.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:28:00.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:28:00.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:28:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:00] "GET /metrics HTTP/1.1" 200 214817 "" "Prometheus/2.33.4" 2024-04-23T03:28:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:01 smithi149 ceph-mon[22716]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:01 smithi007 ceph-mon[21886]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:01 smithi007 ceph-mon[26683]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:03 smithi149 ceph-mon[22716]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:03.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:03 smithi007 ceph-mon[26683]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:03.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:03 smithi007 ceph-mon[21886]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:04 smithi007 ceph-mon[21886]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:04 smithi007 ceph-mon[26683]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:04 smithi149 ceph-mon[22716]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:07 smithi149 ceph-mon[22716]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:07.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:07 smithi007 ceph-mon[21886]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:07 smithi007 ceph-mon[26683]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:07.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:28:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:28:09.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:09 smithi149 ceph-mon[22716]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:09.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:09 smithi007 ceph-mon[21886]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:09.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:09 smithi007 ceph-mon[26683]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:10.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:10 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:10.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:10 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:10.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:10 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:10.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:28:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:10] "GET /metrics HTTP/1.1" 200 214817 "" "Prometheus/2.33.4" 2024-04-23T03:28:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:11 smithi149 ceph-mon[22716]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:11.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:11 smithi007 ceph-mon[26683]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:11.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:11 smithi007 ceph-mon[21886]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:13.379 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:13 smithi149 ceph-mon[22716]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:13.380 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:13 smithi007 ceph-mon[26683]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:13.380 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:13 smithi007 ceph-mon[21886]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:28:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:28:14.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:28:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:28:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:28:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:28:14.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:28:14.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:28:14.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:28:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:15 smithi149 ceph-mon[22716]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:15 smithi007 ceph-mon[26683]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:15 smithi007 ceph-mon[21886]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:17.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:17 smithi149 ceph-mon[22716]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:17.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:17 smithi007 ceph-mon[26683]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:17 smithi007 ceph-mon[21886]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:17.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:28:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:28:19.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:19 smithi149 ceph-mon[22716]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:19.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:19 smithi007 ceph-mon[26683]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:19.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:19 smithi007 ceph-mon[21886]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:20 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:20.490 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:20 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:20.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:20 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:20.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:28:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:20] "GET /metrics HTTP/1.1" 200 214817 "" "Prometheus/2.33.4" 2024-04-23T03:28:21.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:21 smithi149 ceph-mon[22716]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:21.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:21 smithi007 ceph-mon[21886]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:21.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:21 smithi007 ceph-mon[26683]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:23.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:23 smithi149 ceph-mon[22716]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:23 smithi007 ceph-mon[21886]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:23.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:23 smithi007 ceph-mon[26683]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:24 smithi007 ceph-mon[26683]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:24 smithi007 ceph-mon[21886]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:25.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:24 smithi149 ceph-mon[22716]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:27 smithi149 ceph-mon[22716]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:27 smithi007 ceph-mon[26683]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:27 smithi007 ceph-mon[21886]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:27.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:28:27 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:28:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:29 smithi149 ceph-mon[22716]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:29.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:29 smithi007 ceph-mon[26683]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:29 smithi007 ceph-mon[21886]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:30.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:30 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:30.490 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:30 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:30.490 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:30 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:30.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:28:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:30] "GET /metrics HTTP/1.1" 200 214803 "" "Prometheus/2.33.4" 2024-04-23T03:28:31.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:31 smithi149 ceph-mon[22716]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:31 smithi007 ceph-mon[26683]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:31 smithi007 ceph-mon[21886]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:33.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:33 smithi149 ceph-mon[22716]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:33.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:33 smithi007 ceph-mon[26683]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:33.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:33 smithi007 ceph-mon[21886]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:34 smithi007 ceph-mon[26683]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:34 smithi007 ceph-mon[21886]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:34 smithi149 ceph-mon[22716]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:37 smithi149 ceph-mon[22716]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:37.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:37 smithi007 ceph-mon[26683]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:37 smithi007 ceph-mon[21886]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:37.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:28:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:28:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:39 smithi149 ceph-mon[22716]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:39.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:39 smithi007 ceph-mon[26683]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:39 smithi007 ceph-mon[21886]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:40.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:40 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:40.490 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:40 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:40.491 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:40 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:40.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:28:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:40] "GET /metrics HTTP/1.1" 200 214803 "" "Prometheus/2.33.4" 2024-04-23T03:28:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:41 smithi149 ceph-mon[22716]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:41.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:41 smithi007 ceph-mon[21886]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:41 smithi007 ceph-mon[26683]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:43 smithi149 ceph-mon[22716]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:43 smithi007 ceph-mon[26683]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:43 smithi007 ceph-mon[21886]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:45.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:45 smithi007 ceph-mon[26683]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:45.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:45 smithi007 ceph-mon[21886]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:45 smithi149 ceph-mon[22716]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:47 smithi149 ceph-mon[22716]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:47 smithi007 ceph-mon[26683]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:47.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:47 smithi007 ceph-mon[21886]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:47.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:28:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:28:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:49 smithi149 ceph-mon[22716]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:49.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:49 smithi007 ceph-mon[26683]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:49.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:49 smithi007 ceph-mon[21886]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:50 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:50.489 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:50 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:50.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:50 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:28:50.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:28:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:50] "GET /metrics HTTP/1.1" 200 214821 "" "Prometheus/2.33.4" 2024-04-23T03:28:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:51 smithi149 ceph-mon[22716]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:51.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:51 smithi007 ceph-mon[26683]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:51 smithi007 ceph-mon[21886]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:53.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:53 smithi149 ceph-mon[22716]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:53.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:53 smithi007 ceph-mon[26683]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:53.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:53 smithi007 ceph-mon[21886]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:55 smithi007 ceph-mon[26683]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:55.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:55 smithi007 ceph-mon[21886]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:55 smithi149 ceph-mon[22716]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:57 smithi149 ceph-mon[22716]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:57.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:57 smithi007 ceph-mon[26683]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:57 smithi007 ceph-mon[21886]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:28:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:28:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:28:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:28:59 smithi149 ceph-mon[22716]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:28:59 smithi007 ceph-mon[26683]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:28:59.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:28:59 smithi007 ceph-mon[21886]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:29:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:29:00.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:00.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:29:00.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:29:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:00 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:29:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:29:00.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:29:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:00] "GET /metrics HTTP/1.1" 200 214833 "" "Prometheus/2.33.4" 2024-04-23T03:29:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:01 smithi149 ceph-mon[22716]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:01 smithi007 ceph-mon[26683]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:01 smithi007 ceph-mon[21886]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:03 smithi149 ceph-mon[22716]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:03.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:03 smithi007 ceph-mon[26683]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:03.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:03 smithi007 ceph-mon[21886]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:04.357 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-23T03:29:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:05 smithi007 ceph-mon[21886]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:29:05.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:05 smithi007 ceph-mon[26683]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:29:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:05 smithi149 ceph-mon[22716]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:29:06.935 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-23T03:29:07.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:07 smithi007 ceph-mon[26683]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:07.252 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:07 smithi007 ceph-mon[21886]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:07 smithi149 ceph-mon[22716]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:07.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:29:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:29:09.389 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:09 smithi007 ceph-mon[26683]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:29:09.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:09 smithi007 ceph-mon[21886]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:29:09.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-23T03:29:09.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:09 smithi149 ceph-mon[22716]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:29:10.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:10 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:10.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:10 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:10.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:10 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:10.744 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:29:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:10] "GET /metrics HTTP/1.1" 200 214833 "" "Prometheus/2.33.4" 2024-04-23T03:29:11.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:11 smithi007 ceph-mon[26683]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:11 smithi007 ceph-mon[21886]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:11 smithi149 ceph-mon[22716]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:11.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-23T03:29:13.316 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:13 smithi007 ceph-mon[26683]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:13.316 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:13 smithi007 ceph-mon[21886]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:13 smithi149 ceph-mon[22716]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:13.759 INFO:teuthology.orchestra.run.smithi007.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[26683]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[26683]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[21886]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[21886]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:14 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:14.422 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T03:29:14.431 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-23T03:29:14.432 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-23T03:29:14.432 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:29:14.432 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:29:14.432 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:14 smithi149 ceph-mon[22716]: from='client.24958 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:14.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:14 smithi149 ceph-mon[22716]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:14.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:14.433 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:14 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:15 smithi149 ceph-mon[22716]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:15 smithi149 ceph-mon[22716]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:15 smithi007 ceph-mon[21886]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:15 smithi007 ceph-mon[21886]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:15 smithi007 ceph-mon[26683]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:15 smithi007 ceph-mon[26683]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:29:16.821 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:29:17.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:17 smithi149 ceph-mon[22716]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:17.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:17 smithi149 ceph-mon[22716]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:17.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:17 smithi007 ceph-mon[21886]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:17 smithi007 ceph-mon[21886]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:17 smithi007 ceph-mon[26683]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:17 smithi007 ceph-mon[26683]: from='client.15054 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (2m) 2m ago 3m 16.8M - ba2b418f427c 2b72bd483f69 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (2m) 2m ago 2m 47.1M - 8.3.5 dad864ee21e9 2a020aed1bd4 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (2m) 2m ago 2m 72.5M - 3.5 e1d6a67b021e 08dc58563b4c 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443 running (7m) 2m ago 7m 410M - 17.2.0 e1d6a67b021e acee21336e9a 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 1ce9fd79b9c3 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (9m) 2m ago 9m 49.3M 2048M 17.2.0 e1d6a67b021e 06ec9e9186b9 2024-04-23T03:29:17.790 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (7m) 2m ago 7m 38.8M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (8m) 2m ago 8m 39.4M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (3m) 2m ago 3m 17.3M - 1dbe0e931976 e0ac833d4e87 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (3m) 2m ago 3m 15.6M - 1dbe0e931976 519d03321dce 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (7m) 2m ago 7m 52.5M 4017M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (6m) 2m ago 6m 52.8M 4017M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (6m) 2m ago 6m 46.2M 4017M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (6m) 2m ago 6m 49.8M 4017M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (5m) 2m ago 5m 51.2M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (5m) 2m ago 5m 47.2M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (5m) 2m ago 5m 48.9M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (4m) 2m ago 4m 51.8M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (2m) 2m ago 4m 43.9M - 514e6a882f6e fb6f2841055d 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (2m) 2m ago 2m 71.7M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (2m) 2m ago 2m 71.4M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:29:17.791 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (2m) 2m ago 2m 70.4M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:29:17.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:29:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:29:18.345 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:29:18.345 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:29:18.345 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:18.346 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:29:18.347 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-23T03:29:18.347 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:29:18.347 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:29:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:18 smithi149 ceph-mon[22716]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:18 smithi149 ceph-mon[22716]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:18.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:18 smithi007 ceph-mon[26683]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:18.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:18 smithi007 ceph-mon[26683]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:18 smithi007 ceph-mon[21886]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:18 smithi007 ceph-mon[21886]: from='client.34826 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:18.738 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:29:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:29:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:29:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2024-04-23T03:29:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "", 2024-04-23T03:29:18.739 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 image" 2024-04-23T03:29:18.739 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:29:19.218 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:29:19.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:19 smithi149 ceph-mon[22716]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:19.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:19 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1278642500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:29:19.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:19 smithi149 ceph-mon[22716]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:19.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:19 smithi007 ceph-mon[26683]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:19.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:19 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1278642500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:29:19.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:19 smithi007 ceph-mon[26683]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:19.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:19 smithi007 ceph-mon[21886]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:19.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:19 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1278642500' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:29:19.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:19 smithi007 ceph-mon[21886]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:20.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:20 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/735011279' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:29:20.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:20 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:20 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/735011279' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:29:20.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:20 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:20 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/735011279' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:29:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:20 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:20.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:29:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:20] "GET /metrics HTTP/1.1" 200 214818 "" "Prometheus/2.33.4" 2024-04-23T03:29:21.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:21 smithi149 ceph-mon[22716]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:21.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:21 smithi007 ceph-mon[26683]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:21 smithi007 ceph-mon[21886]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:23.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:23 smithi149 ceph-mon[22716]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:23.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:23 smithi007 ceph-mon[26683]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:23 smithi007 ceph-mon[21886]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:25 smithi007 ceph-mon[26683]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:25 smithi007 ceph-mon[21886]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:25.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:25 smithi149 ceph-mon[22716]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:27 smithi149 ceph-mon[22716]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:27.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:27 smithi007 ceph-mon[26683]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:27 smithi007 ceph-mon[21886]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:27.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:29:27 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:29:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:29 smithi149 ceph-mon[22716]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:29 smithi007 ceph-mon[26683]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:29 smithi007 ceph-mon[21886]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:30.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:30 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:30.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:30 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:30.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:30 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:30.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:29:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:30] "GET /metrics HTTP/1.1" 200 214782 "" "Prometheus/2.33.4" 2024-04-23T03:29:31.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:31 smithi149 ceph-mon[22716]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:31 smithi007 ceph-mon[26683]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:31 smithi007 ceph-mon[21886]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:33.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:33 smithi149 ceph-mon[22716]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:33.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:33 smithi007 ceph-mon[26683]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:33.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:33 smithi007 ceph-mon[21886]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:35.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:35 smithi007 ceph-mon[26683]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:35 smithi007 ceph-mon[21886]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:35 smithi149 ceph-mon[22716]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:37 smithi149 ceph-mon[22716]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:37 smithi007 ceph-mon[21886]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:37.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:37 smithi007 ceph-mon[26683]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:37.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:29:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:29:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:39 smithi149 ceph-mon[22716]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:39.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:39 smithi007 ceph-mon[26683]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:39.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:39 smithi007 ceph-mon[21886]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:40.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:40 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:40.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:40 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:40.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:40 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:40.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:29:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:40] "GET /metrics HTTP/1.1" 200 214782 "" "Prometheus/2.33.4" 2024-04-23T03:29:41.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:41 smithi149 ceph-mon[22716]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:41 smithi007 ceph-mon[26683]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:41 smithi007 ceph-mon[21886]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:43.408 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:43 smithi149 ceph-mon[22716]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:43.409 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:43 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:43.409 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:43 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:29:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:43 smithi007 ceph-mon[26683]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:43 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:43 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:29:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:43 smithi007 ceph-mon[21886]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:43 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:43 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:29:44.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:44 smithi149 ceph-mon[22716]: Upgrade: Target is version 19.0.0-3198-g56e81df3 (unknown) 2024-04-23T03:29:44.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:44 smithi149 ceph-mon[22716]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d'] 2024-04-23T03:29:44.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:44 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:44.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:44 smithi149 ceph-mon[22716]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-23T03:29:44.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:44 smithi149 ceph-mon[22716]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 on smithi149 2024-04-23T03:29:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[21886]: Upgrade: Target is version 19.0.0-3198-g56e81df3 (unknown) 2024-04-23T03:29:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[21886]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d'] 2024-04-23T03:29:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[21886]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-23T03:29:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[21886]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 on smithi149 2024-04-23T03:29:44.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[26683]: Upgrade: Target is version 19.0.0-3198-g56e81df3 (unknown) 2024-04-23T03:29:44.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[26683]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, digests ['quay.ceph.io/ceph-ci/ceph@sha256:90c23f6ff09279e806b61a7a0a9b6863a8d239088a6c790da6364c3d6ab9ae8d'] 2024-04-23T03:29:44.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:29:44.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[26683]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-23T03:29:44.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:44 smithi007 ceph-mon[26683]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 on smithi149 2024-04-23T03:29:45.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:45 smithi149 ceph-mon[22716]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:45.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:45 smithi007 ceph-mon[21886]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:45.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:45 smithi007 ceph-mon[26683]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:47.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:47 smithi149 ceph-mon[22716]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:47.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:47 smithi007 ceph-mon[21886]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:47 smithi007 ceph-mon[26683]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:47.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:29:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:29:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:49 smithi149 ceph-mon[22716]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:49.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:49 smithi007 ceph-mon[26683]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:49.560 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:49 smithi007 ceph-mon[21886]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:49.560 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 2m ago 3m 16.8M - ba2b418f427c 2b72bd483f69 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (3m) 2m ago 3m 47.1M - 8.3.5 dad864ee21e9 2a020aed1bd4 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (2m) 2m ago 2m 72.5M - 3.5 e1d6a67b021e 08dc58563b4c 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443 running (8m) 2m ago 8m 410M - 17.2.0 e1d6a67b021e acee21336e9a 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 1ce9fd79b9c3 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (9m) 2m ago 10m 49.3M 2048M 17.2.0 e1d6a67b021e 06ec9e9186b9 2024-04-23T03:29:50.167 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (8m) 2m ago 8m 38.8M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (8m) 2m ago 8m 39.4M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 2m ago 4m 17.3M - 1dbe0e931976 e0ac833d4e87 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (4m) 2m ago 4m 15.6M - 1dbe0e931976 519d03321dce 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (7m) 2m ago 7m 52.5M 4017M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (7m) 2m ago 7m 52.8M 4017M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (7m) 2m ago 7m 46.2M 4017M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (6m) 2m ago 6m 49.8M 4017M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (6m) 2m ago 6m 51.2M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (5m) 2m ago 5m 47.2M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:29:50.168 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (5m) 2m ago 5m 48.9M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:29:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (5m) 2m ago 5m 51.8M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:29:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (3m) 2m ago 4m 43.9M - 514e6a882f6e fb6f2841055d 2024-04-23T03:29:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (3m) 2m ago 3m 70.2M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:29:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (2m) 2m ago 2m 71.7M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:29:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (2m) 2m ago 2m 71.4M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:29:50.169 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (2m) 2m ago 2m 70.4M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:29:50.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-mon[21886]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:50.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-mon[21886]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:50.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:50.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-mon[26683]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:50.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-mon[26683]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:50.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:50 smithi149 ceph-mon[22716]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:50 smithi149 ceph-mon[22716]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:50 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:29:50.529 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:29:50.529 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:29:50.529 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-23T03:29:50.529 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:50.529 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-23T03:29:50.530 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:29:50.531 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:29:50.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:29:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:50] "GET /metrics HTTP/1.1" 200 214751 "" "Prometheus/2.33.4" 2024-04-23T03:29:50.836 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:29:50.836 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:29:50.836 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:29:50.836 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2024-04-23T03:29:50.837 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "0/23 daemons upgraded", 2024-04-23T03:29:50.837 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 image on host smithi149" 2024-04-23T03:29:50.837 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:29:51.206 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:29:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:51 smithi149 ceph-mon[22716]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:51 smithi149 ceph-mon[22716]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:51 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3656538672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:29:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:51 smithi149 ceph-mon[22716]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[26683]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[26683]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3656538672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[26683]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[21886]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[21886]: from='client.25012 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3656538672' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:29:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:51 smithi007 ceph-mon[21886]: from='client.15114 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:29:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:52 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3817766897' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:29:52.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:52 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3817766897' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:29:52.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:52 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3817766897' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:29:53.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:53 smithi149 ceph-mon[22716]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:53.508 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:53 smithi007 ceph-mon[26683]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:53.508 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:53 smithi007 ceph-mon[21886]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:53.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:29:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:29:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:29:53.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:29:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:29:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:29:55.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:55 smithi007 ceph-mon[26683]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:55.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:55 smithi007 ceph-mon[21886]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:55.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:55 smithi149 ceph-mon[22716]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:57 smithi149 ceph-mon[22716]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:57.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:57 smithi007 ceph-mon[26683]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:57 smithi007 ceph-mon[21886]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:29:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:29:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:29:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:29:59 smithi149 ceph-mon[22716]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:29:59 smithi007 ceph-mon[26683]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:29:59.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:29:59 smithi007 ceph-mon[21886]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[26683]: overall HEALTH_OK 2024-04-23T03:30:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:00.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:00.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:00.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[21886]: overall HEALTH_OK 2024-04-23T03:30:00.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:00.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:00 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:00 smithi149 ceph-mon[22716]: overall HEALTH_OK 2024-04-23T03:30:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:00 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:00.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:00] "GET /metrics HTTP/1.1" 200 214754 "" "Prometheus/2.33.4" 2024-04-23T03:30:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:01 smithi149 ceph-mon[22716]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:01 smithi007 ceph-mon[26683]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:01.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:01 smithi007 ceph-mon[21886]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:03 smithi149 ceph-mon[22716]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:03.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:03 smithi007 ceph-mon[26683]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:03.507 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:03 smithi007 ceph-mon[21886]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:03.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:30:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:03.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:03.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:05 smithi007 ceph-mon[26683]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:05 smithi007 ceph-mon[21886]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:05.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:05 smithi149 ceph-mon[22716]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:07 smithi149 ceph-mon[22716]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:07 smithi007 ceph-mon[26683]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:07 smithi007 ceph-mon[21886]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:07.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[23900]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:30:09.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:09 smithi149 ceph-mon[22716]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:09 smithi007 ceph-mon[26683]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:09 smithi007 ceph-mon[21886]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:10.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:10 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:10.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:10 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:10.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:10 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:10.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:10] "GET /metrics HTTP/1.1" 200 214754 "" "Prometheus/2.33.4" 2024-04-23T03:30:11.388 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:11 smithi149 ceph-mon[22716]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:11.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:11 smithi007 ceph-mon[26683]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:11.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:11 smithi007 ceph-mon[21886]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:30:13.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:13.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: Upgrade: Updating mgr.x 2024-04-23T03:30:13.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:13.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:30:13.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:30:13.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[21886]: Deploying daemon mgr.x on smithi149 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: Upgrade: Updating mgr.x 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:30:13.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:13.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-mon[26683]: Deploying daemon mgr.x on smithi149 2024-04-23T03:30:13.585 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:30:13.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:13.585 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:13.585 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: Upgrade: Updating mgr.x 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:13.634 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:13 smithi149 ceph-mon[22716]: Deploying daemon mgr.x on smithi149 2024-04-23T03:30:13.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:13 smithi149 systemd[1]: Stopping Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:30:13.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:13 smithi149 bash[43801]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-23T03:30:14.237 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:13 smithi149 podman[43814]: 2024-04-23 03:30:13.981915631 +0000 UTC m=+0.115775678 container died acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, name=centos-stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, ceph=True, release=754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-23T03:30:14.496 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 podman[43814]: 2024-04-23 03:30:14.319276546 +0000 UTC m=+0.453136590 container cleanup acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, RELEASE=HEAD, release=754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-23T03:30:14.497 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 bash[43814]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x 2024-04-23T03:30:14.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:14.184Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:30:14.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 podman[43828]: 2024-04-23 03:30:14.496023994 +0000 UTC m=+0.510562126 container remove acee21336e9ae4e9e15837ac0ce75799814946e7905599d6445a5e3af40e6e8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, ceph=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.expose-services=, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64) 2024-04-23T03:30:14.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:30:14.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 bash[43840]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-23T03:30:14.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-23T03:30:14.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 systemd[1]: Stopped Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:30:14.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service: Consumed 16.869s CPU time. 2024-04-23T03:30:14.923 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 systemd[1]: Starting Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:30:15.219 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:15 smithi149 ceph-mon[22716]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:15.219 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 podman[43944]: 2024-04-23 03:30:14.86160713 +0000 UTC m=+0.018265027 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:30:15.220 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:14 smithi149 podman[43944]: 2024-04-23 03:30:14.981011246 +0000 UTC m=+0.137669119 container create ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df) 2024-04-23T03:30:15.220 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 podman[43944]: 2024-04-23 03:30:15.191897195 +0000 UTC m=+0.348555068 container init ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-56e81df, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416) 2024-04-23T03:30:15.220 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 podman[43944]: 2024-04-23 03:30:15.194605344 +0000 UTC m=+0.351263220 container start ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True) 2024-04-23T03:30:15.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:15 smithi007 ceph-mon[21886]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:15.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:15 smithi007 ceph-mon[26683]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:15.486 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 ceph-mgr[43963]: -- 172.21.15.149:0/2495957998 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eb90f665a0 con 0x55eb90f3b000 2024-04-23T03:30:15.487 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:15.344+0000 7f6688a1f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:30:15.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 bash[43944]: ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 2024-04-23T03:30:15.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 systemd[1]: Started Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:30:15.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:15.534+0000 7f6688a1f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:30:15.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:15.645+0000 7f6688a1f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:30:16.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:15.996+0000 7f6688a1f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:30:16.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:16 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:16.081+0000 7f6688a1f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:30:16.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:16 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:16.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:16 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:16.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:16 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:30:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:16 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:16 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:16 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:30:16.841 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:16 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:16.841 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:16 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:16.841 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:16 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:30:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:17 smithi149 ceph-mon[22716]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:17.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:17.327+0000 7f6688a1f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:30:17.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:17.502+0000 7f6688a1f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:30:17.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:17 smithi007 ceph-mon[26683]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:17 smithi007 ceph-mon[21886]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:17.981 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:17.682+0000 7f6688a1f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:30:17.981 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:17.979+0000 7f6688a1f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:30:18.299 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:18.077+0000 7f6688a1f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:30:18.299 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:18.290+0000 7f6688a1f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:30:18.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:18.487+0000 7f6688a1f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:30:19.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:19.102+0000 7f6688a1f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:30:19.827 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:19 smithi149 ceph-mon[22716]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:19.828 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:19.528+0000 7f6688a1f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:30:19.828 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:19.625+0000 7f6688a1f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:30:19.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:19 smithi007 ceph-mon[26683]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:19.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:19 smithi007 ceph-mon[21886]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:20.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:19.826+0000 7f6688a1f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T03:30:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:20 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:20.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:20 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:20.364+0000 7f6688a1f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:30:20.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:20 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:20.457+0000 7f6688a1f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:30:20.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:20 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:20 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:20.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:20] "GET /metrics HTTP/1.1" 200 214761 "" "Prometheus/2.33.4" 2024-04-23T03:30:21.022 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:20 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:20.710+0000 7f6688a1f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:30:21.023 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:20 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:20.843+0000 7f6688a1f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:30:21.023 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:20.940+0000 7f6688a1f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:30:21.374 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:21.060+0000 7f6688a1f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:30:21.544 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:30:21.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:21 smithi149 ceph-mon[22716]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:21.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:21.373+0000 7f6688a1f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:30:21.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:21 smithi007 ceph-mon[21886]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:21.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:21 smithi007 ceph-mon[26683]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 3m ago 4m 16.8M - ba2b418f427c 2b72bd483f69 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (4m) 3m ago 4m 47.1M - 8.3.5 dad864ee21e9 2a020aed1bd4 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (3m) 3m ago 3m 72.5M - 3.5 e1d6a67b021e 08dc58563b4c 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283 starting - - - - 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 1ce9fd79b9c3 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (10m) 3m ago 10m 49.3M 2048M 17.2.0 e1d6a67b021e 06ec9e9186b9 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (9m) 3m ago 9m 38.8M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (9m) 3m ago 9m 39.4M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 3m ago 4m 17.3M - 1dbe0e931976 e0ac833d4e87 2024-04-23T03:30:22.150 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (4m) 3m ago 4m 15.6M - 1dbe0e931976 519d03321dce 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (8m) 3m ago 8m 52.5M 4017M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (8m) 3m ago 8m 52.8M 4017M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (7m) 3m ago 7m 46.2M 4017M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (7m) 3m ago 7m 49.8M 4017M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (6m) 3m ago 6m 51.2M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (6m) 3m ago 6m 47.2M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (6m) 3m ago 6m 48.9M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (5m) 3m ago 5m 51.8M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (3m) 3m ago 5m 43.9M - 514e6a882f6e fb6f2841055d 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (3m) 3m ago 3m 70.2M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (3m) 3m ago 3m 71.7M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (3m) 3m ago 3m 71.4M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:30:22.151 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (3m) 3m ago 3m 70.4M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:30:22.325 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:21.994+0000 7f6688a1f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:30:22.325 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:22.216+0000 7f6688a1f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:30:22.325 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:22] ENGINE Bus STARTING 2024-04-23T03:30:22.325 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: CherryPy Checker: 2024-04-23T03:30:22.325 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: The Application mounted at '' has an empty config. 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:30:22.510 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:30:22.511 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:22] ENGINE Serving on http://:::9283 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:22] ENGINE Bus STARTED 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: Standby manager daemon x restarted 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:30:22.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:30:22.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:22 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/409125916' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:30:22.824 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:30:22.824 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:30:22.824 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:30:22.824 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2024-04-23T03:30:22.825 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "", 2024-04-23T03:30:22.825 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mgr daemons" 2024-04-23T03:30:22.825 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: Standby manager daemon x restarted 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:30:22.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/409125916' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='client.15138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: Standby manager daemon x restarted 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: Standby manager daemon x started 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:30:22.826 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:22 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/409125916' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:30:23.190 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:30:23.508 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:23.207Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:23.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[26683]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[26683]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[26683]: mgrmap e22: y(active, since 5m), standbys: x 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[26683]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2110362819' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[21886]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[21886]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[21886]: mgrmap e22: y(active, since 5m), standbys: x 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[21886]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:23.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2110362819' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:30:23.834 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:30:23.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:23.834 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:23.834 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:23.888 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:23 smithi149 ceph-mon[22716]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:23.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:23 smithi149 ceph-mon[22716]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:23.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:23 smithi149 ceph-mon[22716]: mgrmap e22: y(active, since 5m), standbys: x 2024-04-23T03:30:23.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:23 smithi149 ceph-mon[22716]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:23.889 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:23 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2110362819' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:30:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:25 smithi149 ceph-mon[22716]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:25.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:25 smithi007 ceph-mon[26683]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:25 smithi007 ceph-mon[21886]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:27 smithi149 ceph-mon[22716]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:27.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:27 smithi007 ceph-mon[26683]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:27 smithi007 ceph-mon[21886]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:27.851 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:27 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:30:29.379 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:29 smithi149 ceph-mon[22716]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:29.379 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:29 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:29 smithi007 ceph-mon[26683]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:29 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:29 smithi007 ceph-mon[21886]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:29 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:30.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:30] "GET /metrics HTTP/1.1" 200 214773 "" "Prometheus/2.33.4" 2024-04-23T03:30:30.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:30 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:30.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:30 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:30.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:30 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:30.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:30 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:30.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:30 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:30.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:30 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:31.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:31 smithi149 ceph-mon[22716]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:32.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:31 smithi007 ceph-mon[26683]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:32.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:31 smithi007 ceph-mon[21886]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:33.798 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:30:33.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:33.798 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:33.798 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:34.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:34.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:34.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:34.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:30:34.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-23T03:30:34.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: Failing over to other MGR 2024-04-23T03:30:34.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-23T03:30:34.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: osdmap e96: 8 total, 8 up, 8 in 2024-04-23T03:30:34.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[26683]: Standby manager daemon y started 2024-04-23T03:30:34.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: Failing over to other MGR 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: osdmap e96: 8 total, 8 up, 8 in 2024-04-23T03:30:34.085 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mon[21886]: Standby manager daemon y started 2024-04-23T03:30:34.086 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:33.809+0000 7fe838d6e700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-23T03:30:34.086 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setuser ceph since I am not root 2024-04-23T03:30:34.086 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ignoring --setgroup ceph since I am not root 2024-04-23T03:30:34.086 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-mgr[22123]: -- 172.21.15.7:0/1713139035 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555ef473e340 con 0x555ef4716c00 2024-04-23T03:30:34.086 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:33.997+0000 7f372b9af000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:30:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' 2024-04-23T03:30:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: Upgrade: Need to upgrade myself (mgr.y) 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: Failing over to other MGR 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: osdmap e96: 8 total, 8 up, 8 in 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-mon[22716]: Standby manager daemon y started 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:33] ENGINE Bus STOPPING 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:30:34.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:33] ENGINE Bus STOPPED 2024-04-23T03:30:34.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:33] ENGINE Bus STARTING 2024-04-23T03:30:34.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:34] ENGINE Serving on http://:::9283 2024-04-23T03:30:34.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:30:34] ENGINE Bus STARTED 2024-04-23T03:30:34.572 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:34.111Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=2 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:30:34.573 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:34.143+0000 7f372b9af000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:30:34.573 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:34.247+0000 7f372b9af000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:30:34.573 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:34.570+0000 7f372b9af000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:30:34.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:34.656+0000 7f372b9af000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:30:34.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: mgrmap e23: x(active, starting, since 0.999211s), standbys: y 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:30:34.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: Manager daemon x is now available 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: Queued rgw.foo for migration 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: Queued rgw.smpl for migration 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-23T03:30:34.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:30:34.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: mgrmap e23: x(active, starting, since 0.999211s), standbys: y 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:30:34.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: Manager daemon x is now available 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: Queued rgw.foo for migration 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:34.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: Queued rgw.smpl for migration 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:34.839 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:34 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:30:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:30:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.24343 172.21.15.7:0/1923629616' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: mgrmap e23: x(active, starting, since 0.999211s), standbys: y 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:30:35.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: Manager daemon x is now available 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: Queued rgw.foo for migration 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: Queued rgw.smpl for migration 2024-04-23T03:30:35.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:35.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:34 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-23T03:30:36.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:35.847+0000 7f372b9af000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:30:36.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:36.022+0000 7f372b9af000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:30:36.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[26683]: Deploying cephadm binary to smithi149 2024-04-23T03:30:36.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[26683]: Deploying cephadm binary to smithi007 2024-04-23T03:30:36.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[26683]: mgrmap e24: x(active, since 2s), standbys: y 2024-04-23T03:30:36.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[26683]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[21886]: Deploying cephadm binary to smithi149 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[21886]: Deploying cephadm binary to smithi007 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[21886]: mgrmap e24: x(active, since 2s), standbys: y 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[21886]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:36.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:35 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:36.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:35 smithi149 ceph-mon[22716]: Deploying cephadm binary to smithi149 2024-04-23T03:30:36.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:35 smithi149 ceph-mon[22716]: Deploying cephadm binary to smithi007 2024-04-23T03:30:36.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:35 smithi149 ceph-mon[22716]: mgrmap e24: x(active, since 2s), standbys: y 2024-04-23T03:30:36.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:35 smithi149 ceph-mon[22716]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:36.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:35 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:36.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:35 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:36.334 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:36.111+0000 7f372b9af000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:30:36.649 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:36.448+0000 7f372b9af000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:30:36.649 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:36.547+0000 7f372b9af000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:30:36.908 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[26683]: [23/Apr/2024:03:30:35] ENGINE Bus STARTING 2024-04-23T03:30:36.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[26683]: [23/Apr/2024:03:30:35] ENGINE Serving on http://172.21.15.149:8765 2024-04-23T03:30:36.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[26683]: [23/Apr/2024:03:30:35] ENGINE Serving on https://172.21.15.149:7150 2024-04-23T03:30:36.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[26683]: [23/Apr/2024:03:30:35] ENGINE Bus STARTED 2024-04-23T03:30:36.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[26683]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:36.909 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:36.647+0000 7f372b9af000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:30:36.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[21886]: [23/Apr/2024:03:30:35] ENGINE Bus STARTING 2024-04-23T03:30:36.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[21886]: [23/Apr/2024:03:30:35] ENGINE Serving on http://172.21.15.149:8765 2024-04-23T03:30:36.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[21886]: [23/Apr/2024:03:30:35] ENGINE Serving on https://172.21.15.149:7150 2024-04-23T03:30:36.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[21886]: [23/Apr/2024:03:30:35] ENGINE Bus STARTED 2024-04-23T03:30:36.910 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-mon[21886]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:36 smithi149 ceph-mon[22716]: [23/Apr/2024:03:30:35] ENGINE Bus STARTING 2024-04-23T03:30:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:36 smithi149 ceph-mon[22716]: [23/Apr/2024:03:30:35] ENGINE Serving on http://172.21.15.149:8765 2024-04-23T03:30:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:36 smithi149 ceph-mon[22716]: [23/Apr/2024:03:30:35] ENGINE Serving on https://172.21.15.149:7150 2024-04-23T03:30:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:36 smithi149 ceph-mon[22716]: [23/Apr/2024:03:30:35] ENGINE Bus STARTED 2024-04-23T03:30:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:36 smithi149 ceph-mon[22716]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:37.299 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:36.907+0000 7f372b9af000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:30:37.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:37.379+0000 7f372b9af000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:30:37.827 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:37] "GET /metrics HTTP/1.1" 200 35342 "" "Prometheus/2.33.4" 2024-04-23T03:30:38.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:37 smithi007 ceph-mon[26683]: mgrmap e25: x(active, since 4s), standbys: y 2024-04-23T03:30:38.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:37 smithi007 ceph-mon[21886]: mgrmap e25: x(active, since 4s), standbys: y 2024-04-23T03:30:38.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:37.890+0000 7f372b9af000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:30:38.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:37 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:37.989+0000 7f372b9af000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:30:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:37 smithi149 ceph-mon[22716]: mgrmap e25: x(active, since 4s), standbys: y 2024-04-23T03:30:38.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:38.299+0000 7f372b9af000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:30:38.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:38.393+0000 7f372b9af000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:30:38.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:38 smithi149 ceph-mon[22716]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:30:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:30:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: |\n OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked down and back up at {{ $value | humanize }} times once a minute for 5 minutes. This could indicate a network issue (latency, packet drop, disruption) on the clusters \"cluster network\". Check the network environment on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSD's to flap (mark each other out)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.7:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:30:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:30:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:30:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephPGImbalance\nexpr: abs(((ceph_osd_numpg > 0) - on(job) group_left() avg by(job) (ceph_osd_numpg\n > 0)) / on(job) group_left() avg by(job) (ceph_osd_numpg > 0)) * on(ceph_daemon)\n group_left(hostname) ceph_osd_metadata > 0.3\nfor: 5m\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.5\n severity: warning\n type: ceph_default\nannotations:\n description: |\n OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} deviates by more than 30% from average PG count.\n summary: PG allocations are not balanced across devices\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.7:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:30:39.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:38 smithi007 ceph-mon[21886]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:38 smithi007 ceph-mon[26683]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:39.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:38.815+0000 7f372b9af000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:30:39.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:38.955+0000 7f372b9af000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:30:39.083 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:39 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:39.054+0000 7f372b9af000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:30:39.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:39 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:39.179+0000 7f372b9af000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:30:39.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:39 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:39.276+0000 7f372b9af000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:30:40.273 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:39 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:39.875+0000 7f372b9af000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:30:40.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:40.271+0000 7f372b9af000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:30:40] ENGINE Bus STARTING 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: CherryPy Checker: 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: The Application mounted at '' has an empty config. 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:30:40] ENGINE Serving on http://:::9283 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: [23/Apr/2024:03:30:40] ENGINE Bus STARTED 2024-04-23T03:30:40.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:30:40.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:40.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:40.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:40.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:40.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: Standby manager daemon y restarted 2024-04-23T03:30:40.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: Standby manager daemon y started 2024-04-23T03:30:40.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T03:30:40.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:30:40.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T03:30:40.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:40 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:30:41.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:41.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: Standby manager daemon y restarted 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: Standby manager daemon y started 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:41.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: Standby manager daemon y restarted 2024-04-23T03:30:41.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: Standby manager daemon y started 2024-04-23T03:30:41.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T03:30:41.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:30:41.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T03:30:41.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:40 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/292085553' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:30:41.643 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:41 smithi149 ceph-mon[22716]: mgrmap e26: x(active, since 7s), standbys: y 2024-04-23T03:30:41.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:41 smithi007 ceph-mon[26683]: mgrmap e26: x(active, since 7s), standbys: y 2024-04-23T03:30:41.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:41 smithi007 ceph-mon[21886]: mgrmap e26: x(active, since 7s), standbys: y 2024-04-23T03:30:42.210 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:41 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:41.877Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=6 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:42.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:42 smithi149 ceph-mon[22716]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-23T03:30:42.964 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:42 smithi007 ceph-mon[26683]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-23T03:30:42.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:42 smithi007 ceph-mon[21886]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-23T03:30:43.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:43 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:30:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:43.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:43 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:43.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:43 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:44.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[26683]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:44 smithi007 ceph-mon[21886]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-23T03:30:44.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:44 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:44 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:44 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:44 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:44.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:44 smithi149 ceph-mon[22716]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-23T03:30:45.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi007 to 2737M 2024-04-23T03:30:45.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi007 to 2737M 2024-04-23T03:30:45.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:45 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi007 to 2737M 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:45.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:45 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:46 smithi149 ceph-mon[22716]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:30:47.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:46 smithi007 ceph-mon[26683]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:30:47.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:46 smithi007 ceph-mon[21886]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:30:47.591 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:47] "GET /metrics HTTP/1.1" 200 35342 "" "Prometheus/2.33.4" 2024-04-23T03:30:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:49.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:49.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.336 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.337 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:48 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:30:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:30:49.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:30:49.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:49.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:48 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:30:49.984 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:49 smithi007 systemd[1]: Stopping Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:30:49.984 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:49 smithi007 bash[55725]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-23T03:30:49.984 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:49 smithi007 podman[55737]: 2024-04-23 03:30:49.765241114 +0000 UTC m=+0.118933323 container died e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[26683]: Reconfiguring daemon node-exporter.a on smithi007 2024-04-23T03:30:50.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.250 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-23T03:30:50.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-mon[21886]: Reconfiguring daemon node-exporter.a on smithi007 2024-04-23T03:30:50.251 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 podman[55737]: 2024-04-23 03:30:50.205700351 +0000 UTC m=+0.559392620 container cleanup e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:50.251 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 bash[55737]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a 2024-04-23T03:30:50.251 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 bash[55763]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-23T03:30:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:30:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:30:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:30:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:30:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:30:50.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Reconfiguring node-exporter.a (dependencies changed)... 2024-04-23T03:30:50.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:50 smithi149 ceph-mon[22716]: Reconfiguring daemon node-exporter.a on smithi007 2024-04-23T03:30:50.758 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:30:50.758 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 podman[55751]: 2024-04-23 03:30:50.498678469 +0000 UTC m=+0.737206104 container remove e0ac833d4e87dc181df603c3982198b36c3b7f0b6d19ab6f85f77f0e94269a3b (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 systemd[1]: Stopped Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 2.828s CPU time. 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 systemd[1]: Starting Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 podman[55833]: 2024-04-23 03:30:50.640956988 +0000 UTC m=+0.018142877 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-23T03:30:50.759 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 podman[55833]: 2024-04-23 03:30:50.758127566 +0000 UTC m=+0.135313504 container create 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:51.078 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 podman[55833]: 2024-04-23 03:30:50.977284736 +0000 UTC m=+0.354470637 container init 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:51.078 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 podman[55833]: 2024-04-23 03:30:50.980090783 +0000 UTC m=+0.357276671 container start 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:51.078 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.983Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.983Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.984Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.984Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=arp 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-23T03:30:51.079 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=edac 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-23T03:30:51.080 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-23T03:30:51.081 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=os 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=stat 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-23T03:30:51.082 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=time 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=uname 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a[55847]: ts=2024-04-23T03:30:50.985Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-23T03:30:51.083 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:51 smithi007 bash[55833]: 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f 2024-04-23T03:30:51.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:51 smithi007 ceph-mon[26683]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:51.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:51 smithi007 ceph-mon[26683]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:51 smithi007 ceph-mon[21886]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:51.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:51 smithi007 ceph-mon[21886]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:51.333 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:30:51 smithi007 systemd[1]: Started Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:30:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:51 smithi149 ceph-mon[22716]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:51 smithi149 ceph-mon[22716]: from='client.24802 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:30:52.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:52.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:52.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[26683]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-23T03:30:52.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[26683]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-23T03:30:52.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:52.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:52.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[21886]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-23T03:30:52.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:52 smithi007 ceph-mon[21886]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-23T03:30:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:52 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:52 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:52 smithi149 ceph-mon[22716]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-23T03:30:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:52 smithi149 ceph-mon[22716]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-23T03:30:52.422 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:30:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:30:52.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.7:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:30:53.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:53 smithi149 ceph-mon[22716]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:53.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:53 smithi007 ceph-mon[26683]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:53.497 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:53 smithi007 ceph-mon[21886]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:30:53.563 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:30:53.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=error ts=2024-04-23T03:30:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:53.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-23T03:30:53.833 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:53 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=warn ts=2024-04-23T03:30:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.149:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.149 because it doesn't contain any IP SANs" 2024-04-23T03:30:54.265 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:30:54.265 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (4m) 10s ago 4m 26.7M - 0.23.0 ba2b418f427c 2b72bd483f69 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (4m) 10s ago 4m 56.1M - 8.3.5 dad864ee21e9 2a020aed1bd4 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (3m) 10s ago 3m 53.6M - 3.5 e1d6a67b021e 08dc58563b4c 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283 running (39s) 10s ago 9m 493M - 19.0.0-3198-g56e81df3 82086b19bb07 ff28fa5468e0 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (10m) 10s ago 10m 395M - 17.2.0 e1d6a67b021e 1ce9fd79b9c3 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (11m) 10s ago 11m 55.9M 2048M 17.2.0 e1d6a67b021e 06ec9e9186b9 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (9m) 10s ago 9m 41.8M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (9m) 10s ago 9m 40.9M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 starting - - - - 2024-04-23T03:30:54.266 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (5m) 10s ago 5m 15.9M - 1.3.1 1dbe0e931976 519d03321dce 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (9m) 10s ago 8m 55.4M 2737M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (8m) 10s ago 8m 56.0M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (8m) 10s ago 8m 47.8M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (7m) 10s ago 7m 52.9M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (7m) 10s ago 7m 54.9M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (7m) 10s ago 7m 50.2M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (6m) 10s ago 6m 50.7M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (6m) 10s ago 6m 55.4M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (4m) 10s ago 5m 65.3M - 2.33.4 514e6a882f6e fb6f2841055d 2024-04-23T03:30:54.267 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (4m) 10s ago 4m 71.9M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:30:54.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (4m) 10s ago 4m 74.3M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:30:54.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (3m) 10s ago 3m 73.3M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:30:54.268 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (3m) 10s ago 3m 73.1M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 1 2024-04-23T03:30:54.695 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 1 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:30:54.696 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:30:54.733 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:54 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:30:54.733 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:54 smithi007 bash[56113]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-23T03:30:55.043 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:54 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[46587]: level=info ts=2024-04-23T03:30:54.733Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:30:55.044 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:54 smithi007 podman[56126]: 2024-04-23 03:30:54.784213036 +0000 UTC m=+0.175735123 container died 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:55.048 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:30:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:30:55.048 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:30:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:30:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2024-04-23T03:30:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "1/23 daemons upgraded", 2024-04-23T03:30:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "", 2024-04-23T03:30:55.049 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:30:55.049 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:30:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[21886]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[21886]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[21886]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:55.319 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/365220778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:30:55.319 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[26683]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:55.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[26683]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:55.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[26683]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:55.320 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:55 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/365220778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:30:55.320 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 podman[56126]: 2024-04-23 03:30:55.125635443 +0000 UTC m=+0.517157521 container cleanup 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:55.320 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 bash[56126]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a 2024-04-23T03:30:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:55 smithi149 ceph-mon[22716]: from='client.25108 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:55 smithi149 ceph-mon[22716]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:55 smithi149 ceph-mon[22716]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:55 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/365220778' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:30:55.468 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:30:55.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 podman[56143]: 2024-04-23 03:30:55.318587737 +0000 UTC m=+0.530726482 container remove 2b72bd483f692f6cfa386b3ed38279407fa89938a69bdc6e2e0a19a4af18c27d (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:55.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 podman[56143]: 2024-04-23 03:30:55.44523982 +0000 UTC m=+0.657378567 volume remove ace53e1aad9216914ebd23fa5b3c58b1664e61d8064abf40c0af6fecdd7c3862 2024-04-23T03:30:55.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 bash[56179]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-23T03:30:55.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-23T03:30:55.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:30:55.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 systemd[1]: Starting Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:30:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 podman[56271]: 2024-04-23 03:30:55.618653367 +0000 UTC m=+0.018254368 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-23T03:30:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 podman[56271]: 2024-04-23 03:30:55.729660878 +0000 UTC m=+0.129261852 volume create c8135ff41fd7feacfb1aa6b6a909e1688a750075c7f881d27c26cf2dc37848ae 2024-04-23T03:30:56.012 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:55 smithi007 podman[56271]: 2024-04-23 03:30:55.813395545 +0000 UTC m=+0.212996526 container create dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:56.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-mon[26683]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:56.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-mon[26683]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:56.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/4058584498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:30:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-mon[21886]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-mon[21886]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:56.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/4058584498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:30:56.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 podman[56271]: 2024-04-23 03:30:56.124983046 +0000 UTC m=+0.524584020 container init dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:56.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 podman[56271]: 2024-04-23 03:30:56.127743424 +0000 UTC m=+0.527344407 container start dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:30:56.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.140Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-23T03:30:56.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.141Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-23T03:30:56.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.142Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.143Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.168Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.168Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.171Z caller=main.go:518 msg=Listening address=:9093 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:56.171Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 bash[56271]: dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a 2024-04-23T03:30:56.334 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:56 smithi007 systemd[1]: Started Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:30:56.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:56 smithi149 ceph-mon[22716]: from='client.25120 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:56 smithi149 ceph-mon[22716]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:30:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:56 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4058584498' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:30:57.137 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[21886]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-mon[22716]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:30:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:30:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[26683]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:30:57.433 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:30:57.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:57 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:30:57.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:30:57] "GET /metrics HTTP/1.1" 200 38148 "" "Prometheus/2.33.4" 2024-04-23T03:30:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:58 smithi007 ceph-mon[21886]: Reconfiguring iscsi.foo.smithi007.xwmqjz (dependencies changed)... 2024-04-23T03:30:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:58 smithi007 ceph-mon[21886]: Reconfiguring daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:30:58.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:58 smithi007 ceph-mon[26683]: Reconfiguring iscsi.foo.smithi007.xwmqjz (dependencies changed)... 2024-04-23T03:30:58.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:58 smithi007 ceph-mon[26683]: Reconfiguring daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:30:58.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:30:58 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:30:58.143Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000089066s 2024-04-23T03:30:58.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:58 smithi149 ceph-mon[22716]: Reconfiguring iscsi.foo.smithi007.xwmqjz (dependencies changed)... 2024-04-23T03:30:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:58 smithi149 ceph-mon[22716]: Reconfiguring daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:30:59.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:30:59 smithi007 ceph-mon[26683]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:59.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:30:59 smithi007 ceph-mon[21886]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:30:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-mon[22716]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:00.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:30:59.848+0000 7f660e5a0700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:00.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Creating ceph-iscsi config... 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Traceback (most recent call last): 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return _run_code(code, main_globals, None, 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: exec(code, run_globals) 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:00.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Traceback (most recent call last): 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:00.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return future.result(timeout) 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return self.__get_result() 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: raise self._exception 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: use_current_daemon_image=reconfig, 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:00.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Creating ceph-iscsi config... 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Traceback (most recent call last): 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return _run_code(code, main_globals, None, 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: exec(code, run_globals) 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:00.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:00.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:00.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:00.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:30:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:00.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:00 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[22119]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:31:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:31:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:01 smithi149 ceph-mon[22716]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:01 smithi149 ceph-mon[22716]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:01.423 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: return future.result(timeout) 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:01.424 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.__get_result() 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: raise self._exception 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_current_daemon_image=reconfig, 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.425 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.426 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:01 smithi149 ceph-mon[22716]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:31:01.427 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:01 smithi149 ceph-mon[22716]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[26683]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[26683]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.584 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: return future.result(timeout) 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: return self.__get_result() 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: raise self._exception 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:01.585 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_current_daemon_image=reconfig, 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.586 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[21886]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:01.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[21886]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:01.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:01.589 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: return future.result(timeout) 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.__get_result() 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: raise self._exception 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_current_daemon_image=reconfig, 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:01.590 INFO:journalctl@ceph.mon.a.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:01.591 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[21886]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[21886]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:31:01.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:01.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:01.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:01.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:01.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:01.593 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[26683]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:31:01.593 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:01 smithi007 ceph-mon[26683]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:31:02.407 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:02.315Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.7:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.149:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:03.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-mon[22716]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:03.422 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 systemd[1]: Stopping Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:03.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 bash[47307]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-23T03:31:03.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:03 smithi007 ceph-mon[26683]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:03.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:03 smithi007 ceph-mon[21886]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:03.849 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.539Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.539Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.539Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.539Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.539Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.539Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.540Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-23T03:31:03.850 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.540Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-23T03:31:03.851 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.540Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-23T03:31:03.851 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.540Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-23T03:31:03.851 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.540Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-23T03:31:03.851 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[39639]: ts=2024-04-23T03:31:03.540Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-23T03:31:03.851 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 podman[47321]: 2024-04-23 03:31:03.549911848 +0000 UTC m=+0.144237646 container died fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:04.154 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 podman[47321]: 2024-04-23 03:31:03.956188946 +0000 UTC m=+0.550514749 container cleanup fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:04.154 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 bash[47321]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a 2024-04-23T03:31:04.154 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:03 smithi149 bash[47348]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-23T03:31:04.411 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:04.411 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 podman[47335]: 2024-04-23 03:31:04.182662046 +0000 UTC m=+0.635649955 container remove fb6f2841055d789a45f59f434251924ffc4cb68cc42ab5fd1e757b15a58b90e9 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:04.411 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-23T03:31:04.412 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 systemd[1]: Stopped Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:04.412 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a.service: Consumed 1.471s CPU time. 2024-04-23T03:31:04.412 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 systemd[1]: Starting Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:04.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:04 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:04 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:04.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 podman[47415]: 2024-04-23 03:31:04.31473727 +0000 UTC m=+0.019436025 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-23T03:31:04.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 podman[47415]: 2024-04-23 03:31:04.439619245 +0000 UTC m=+0.144317986 container create 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 podman[47415]: 2024-04-23 03:31:04.677360231 +0000 UTC m=+0.382058973 container init 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 podman[47415]: 2024-04-23 03:31:04.680037099 +0000 UTC m=+0.384735840 container start 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.703Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.703Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.703Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.703Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi149 (none))" 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.703Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.703Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.704Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.705Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-23T03:31:05.158 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.706Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-23T03:31:05.159 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.708Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-23T03:31:05.159 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.708Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.64µs 2024-04-23T03:31:05.159 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.708Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-23T03:31:05.159 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 bash[47415]: 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 2024-04-23T03:31:05.159 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 systemd[1]: Started Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:05.159 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:04.923Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-23T03:31:05.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:05 smithi149 systemd[1]: Stopping Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:05.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:05 smithi149 bash[47539]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-23T03:31:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-mon[22716]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:31:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:05.422 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.172Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.172Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.172Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=31.054µs wal_replay_duration=464.187915ms total_replay_duration=464.249338ms 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.176Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.176Z caller=main.go:947 level=info msg="TSDB started" 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.176Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.201Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=24.839019ms db_storage=801ns remote_storage=1.146µs web_handler=410ns query_engine=589ns scrape=375.814µs scrape_sd=107.552µs notify=9.958µs notify_sd=8.676µs rules=23.983289ms 2024-04-23T03:31:05.423 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:05.201Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-23T03:31:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:05 smithi007 ceph-mon[26683]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:31:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:05 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:05 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:05 smithi007 ceph-mon[21886]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:31:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:05 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:05 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:05.798 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:05 smithi149 podman[47552]: 2024-04-23 03:31:05.532976542 +0000 UTC m=+0.144789082 container died 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:31:06.161 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:05 smithi149 podman[47552]: 2024-04-23 03:31:05.897470654 +0000 UTC m=+0.509283189 container cleanup 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:31:06.161 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:05 smithi149 bash[47552]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b 2024-04-23T03:31:06.162 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:05 smithi149 bash[47579]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-23T03:31:06.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-mon[22716]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-23T03:31:06.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-mon[22716]: Reconfiguring daemon node-exporter.b on smithi149 2024-04-23T03:31:06.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 podman[47566]: 2024-04-23 03:31:06.207421555 +0000 UTC m=+0.674606977 container remove 519d03321dce65f78527c5a84b982b3aeb4a664892bd47f2583cf0c6da24efed (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:31:06.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:31:06.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-23T03:31:06.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 systemd[1]: Stopped Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:06.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 2.844s CPU time. 2024-04-23T03:31:06.422 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 systemd[1]: Starting Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:06.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:06 smithi007 ceph-mon[26683]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-23T03:31:06.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:06 smithi007 ceph-mon[26683]: Reconfiguring daemon node-exporter.b on smithi149 2024-04-23T03:31:06.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:06 smithi007 ceph-mon[21886]: Reconfiguring node-exporter.b (dependencies changed)... 2024-04-23T03:31:06.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:06 smithi007 ceph-mon[21886]: Reconfiguring daemon node-exporter.b on smithi149 2024-04-23T03:31:06.484 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:31:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:31:06.146Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002805189s 2024-04-23T03:31:06.702 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 podman[47647]: 2024-04-23 03:31:06.424821507 +0000 UTC m=+0.106320025 container create 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:31:06.702 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 podman[47647]: 2024-04-23 03:31:06.336440793 +0000 UTC m=+0.017939275 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-23T03:31:06.702 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 podman[47647]: 2024-04-23 03:31:06.618724451 +0000 UTC m=+0.300222921 container init 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 podman[47647]: 2024-04-23 03:31:06.621275388 +0000 UTC m=+0.302773858 container start 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.624Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.624Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=arp 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-23T03:31:06.703 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=edac 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-23T03:31:06.704 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=os 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-23T03:31:06.705 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=stat 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=time 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=uname 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.625Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b[47662]: ts=2024-04-23T03:31:06.626Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-23T03:31:06.706 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 bash[47647]: 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 2024-04-23T03:31:07.167 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:31:06 smithi149 systemd[1]: Started Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:07.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:07 smithi149 ceph-mon[22716]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:31:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:07 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:07 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:07 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:07 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:07 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:31:07.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[26683]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[21886]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:07 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:31:08.343 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:08 smithi149 ceph-mon[22716]: Reconfiguring grafana.a (dependencies changed)... 2024-04-23T03:31:08.344 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:08 smithi149 ceph-mon[22716]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-23T03:31:08.344 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:08 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:31:08.344 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:08 smithi149 ceph-mon[22716]: Reconfiguring daemon grafana.a on smithi149 2024-04-23T03:31:08.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[21886]: Reconfiguring grafana.a (dependencies changed)... 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[21886]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[21886]: Reconfiguring daemon grafana.a on smithi149 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[26683]: Reconfiguring grafana.a (dependencies changed)... 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[26683]: Regenerating cephadm self-signed grafana TLS certificates 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-23T03:31:08.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:08 smithi007 ceph-mon[26683]: Reconfiguring daemon grafana.a on smithi149 2024-04-23T03:31:09.443 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 systemd[1]: Stopping Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:09.443 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:09 smithi149 ceph-mon[22716]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-23T03:31:09.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:09 smithi007 ceph-mon[21886]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-23T03:31:09.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:09 smithi007 ceph-mon[26683]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-23T03:31:09.846 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 bash[47836]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-23T03:31:09.846 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:31:09+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-23T03:31:09.847 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[38843]: t=2024-04-23T03:31:09+0000 lvl=info msg="Database locked, sleeping then retrying" logger=sqlstore error="database is locked" retry=0 2024-04-23T03:31:09.847 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 podman[47849]: 2024-04-23 03:31:09.598459367 +0000 UTC m=+0.147472113 container died 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, maintainer=Paul Cuzner , io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, vcs-type=git, com.redhat.component=ubi8-container, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, version=8.5, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public) 2024-04-23T03:31:10.172 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 podman[47849]: 2024-04-23 03:31:09.945148791 +0000 UTC m=+0.494161538 container cleanup 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, com.redhat.component=ubi8-container, release=236.1648460182, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, architecture=x86_64, name=ubi8, version=8.5, io.buildah.version=1.24.2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, io.openshift.tags=base rhel8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI) 2024-04-23T03:31:10.172 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 bash[47849]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a 2024-04-23T03:31:10.172 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:09 smithi149 bash[47876]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-23T03:31:10.464 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 podman[47863]: 2024-04-23 03:31:10.246463587 +0000 UTC m=+0.649923159 container remove 2a020aed1bd4bdb4f5a368cb957145529b0d46350b5a79b0c21b03f5cf9a5d31 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, description=Ceph Grafana Container, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vcs-type=git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, architecture=x86_64, io.buildah.version=1.24.2, release=236.1648460182, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8) 2024-04-23T03:31:10.464 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-23T03:31:10.464 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 systemd[1]: Stopped Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:10.465 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@grafana.a.service: Consumed 2.509s CPU time. 2024-04-23T03:31:10.465 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 systemd[1]: Starting Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:10.723 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 podman[47944]: 2024-04-23 03:31:10.463875608 +0000 UTC m=+0.103306017 container create 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, version=8.5, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, release=236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.buildah.version=1.24.2, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, description=Ceph Grafana Container) 2024-04-23T03:31:10.723 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 podman[47944]: 2024-04-23 03:31:10.377887558 +0000 UTC m=+0.017317967 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-23T03:31:10.723 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 podman[47944]: 2024-04-23 03:31:10.691314646 +0000 UTC m=+0.330745056 container init 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.24.2, name=ubi8, io.openshift.tags=base rhel8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.openshift.expose-services=, description=Ceph Grafana Container, architecture=x86_64) 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 podman[47944]: 2024-04-23 03:31:10.694093289 +0000 UTC m=+0.333523699 container start 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, io.openshift.tags=base rhel8, description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, vendor=Red Hat, Inc., architecture=x86_64, vcs-type=git, release=236.1648460182, io.buildah.version=1.24.2, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public) 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-23T03:31:10.724 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="App mode production" logger=settings 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-23T03:31:10.725 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-23T03:31:10.987 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STOPPING 2024-04-23T03:31:10.987 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:31:10.987 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STOPPED 2024-04-23T03:31:10.987 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STARTING 2024-04-23T03:31:10.988 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Serving on http://:::9283 2024-04-23T03:31:10.988 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STARTED 2024-04-23T03:31:10.988 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-23T03:31:10.988 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=1.031609ms 2024-04-23T03:31:10.988 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-23T03:31:10.988 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-23T03:31:10.988 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 bash[47944]: 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 systemd[1]: Started Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-23T03:31:10.989 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-04-23T03:31:10.990 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-23T03:31:10.990 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-23T03:31:10.990 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:31:10+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-23T03:31:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-23T03:31:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:31:11.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149:9095"}]: dispatch 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149:3000"}]: dispatch 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STOPPING 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STOPPED 2024-04-23T03:31:11.423 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:10] ENGINE Bus STARTING 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Serving on http://:::9283 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Bus STARTED 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Bus STOPPING 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Bus STOPPED 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Bus STARTING 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Serving on http://:::9283 2024-04-23T03:31:11.424 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:11] ENGINE Bus STARTED 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149:9095"}]: dispatch 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149:3000"}]: dispatch 2024-04-23T03:31:11.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149:9095"}]: dispatch 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:31:11.441 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149:3000"}]: dispatch 2024-04-23T03:31:11.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:11.442 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:11 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:12.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:31:12.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-04-23T03:31:12.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:12.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149:9095"}]: dispatch 2024-04-23T03:31:12.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:31:12.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149:3000"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/921487120' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3000731606' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/782590966"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149:9095"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149:3000"}]: dispatch 2024-04-23T03:31:12.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:12.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:12.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/921487120' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:31:12.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:12 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3000731606' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/782590966"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi007.front.sepia.ceph.com:9093"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149:9095"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi149:3000"}]: dispatch 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:12.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/921487120' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:31:12.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:12 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3000731606' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/782590966"}]: dispatch 2024-04-23T03:31:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3000731606' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/782590966"}]': finished 2024-04-23T03:31:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: osdmap e97: 8 total, 8 up, 8 in 2024-04-23T03:31:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1889464340' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]: dispatch 2024-04-23T03:31:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]: dispatch 2024-04-23T03:31:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:31:13.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:13.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:13.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3000731606' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/782590966"}]': finished 2024-04-23T03:31:13.454 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: osdmap e97: 8 total, 8 up, 8 in 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1889464340' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]: dispatch 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]: dispatch 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3000731606' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/782590966"}]': finished 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: osdmap e97: 8 total, 8 up, 8 in 2024-04-23T03:31:13.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1889464340' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]: dispatch 2024-04-23T03:31:13.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]: dispatch 2024-04-23T03:31:13.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:13.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:13.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:13.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:31:13.456 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:13 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:13.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi149\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi149\", job=\"node\", machine=\"x86_64\", nodename=\"smithi149\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi149\", job=\"node\", machine=\"x86_64\", nodename=\"smithi149\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[26683]: Upgrade: Updating mgr.y 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[26683]: Deploying daemon mgr.y on smithi007 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]': finished 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[26683]: osdmap e98: 8 total, 8 up, 8 in 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/889099523' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3451088861"}]: dispatch 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[21886]: Upgrade: Updating mgr.y 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[21886]: Deploying daemon mgr.y on smithi007 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]': finished 2024-04-23T03:31:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[21886]: osdmap e98: 8 total, 8 up, 8 in 2024-04-23T03:31:14.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:14 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/889099523' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3451088861"}]: dispatch 2024-04-23T03:31:14.653 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:14 smithi007 systemd[1]: Stopping Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:14 smithi149 ceph-mon[22716]: Upgrade: Updating mgr.y 2024-04-23T03:31:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:14 smithi149 ceph-mon[22716]: Deploying daemon mgr.y on smithi007 2024-04-23T03:31:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:14 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/541957860"}]': finished 2024-04-23T03:31:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:14 smithi149 ceph-mon[22716]: osdmap e98: 8 total, 8 up, 8 in 2024-04-23T03:31:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:14 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/889099523' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3451088861"}]: dispatch 2024-04-23T03:31:14.964 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:14 smithi007 podman[57171]: 2024-04-23 03:31:14.652202564 +0000 UTC m=+0.143347521 container died 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, ceph=True, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, RELEASE=HEAD) 2024-04-23T03:31:15.218 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[26683]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:15.218 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/889099523' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3451088861"}]': finished 2024-04-23T03:31:15.218 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[26683]: osdmap e99: 8 total, 8 up, 8 in 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2606963688' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2317307583"}]: dispatch 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[21886]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/889099523' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3451088861"}]': finished 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[21886]: osdmap e99: 8 total, 8 up, 8 in 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:15 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2606963688' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2317307583"}]: dispatch 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 podman[57171]: 2024-04-23 03:31:15.171729413 +0000 UTC m=+0.662874377 container cleanup 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, architecture=x86_64, release=754, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vcs-type=git) 2024-04-23T03:31:15.219 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 bash[57171]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y 2024-04-23T03:31:15.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 podman[57186]: 2024-04-23 03:31:15.423771151 +0000 UTC m=+0.776132209 container remove 1ce9fd79b9c39391c2773ac0dbde5677035fead7c0d50e515d73f164456f73cf (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, architecture=x86_64, vcs-type=git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8) 2024-04-23T03:31:15.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:31:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:15 smithi149 ceph-mon[22716]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:15 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/889099523' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3451088861"}]': finished 2024-04-23T03:31:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:15 smithi149 ceph-mon[22716]: osdmap e99: 8 total, 8 up, 8 in 2024-04-23T03:31:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:15 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2606963688' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2317307583"}]: dispatch 2024-04-23T03:31:16.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-04-23T03:31:16.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 systemd[1]: Stopped Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:16.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service: Consumed 58.314s CPU time. 2024-04-23T03:31:16.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:15 smithi007 systemd[1]: Starting Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/2606963688' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2317307583"}]': finished 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[26683]: osdmap e100: 8 total, 8 up, 8 in 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1848217604' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]: dispatch 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]: dispatch 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/2606963688' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2317307583"}]': finished 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[21886]: osdmap e100: 8 total, 8 up, 8 in 2024-04-23T03:31:16.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1848217604' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]: dispatch 2024-04-23T03:31:16.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]: dispatch 2024-04-23T03:31:16.373 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 podman[57341]: 2024-04-23 03:31:16.009664615 +0000 UTC m=+0.018804814 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:31:16.373 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 podman[57341]: 2024-04-23 03:31:16.141832811 +0000 UTC m=+0.150973037 container create 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T03:31:16.373 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 podman[57341]: 2024-04-23 03:31:16.344669988 +0000 UTC m=+0.353810173 container init 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:31:16.373 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 podman[57341]: 2024-04-23 03:31:16.347432924 +0000 UTC m=+0.356573112 container start 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-23T03:31:16.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:16 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/2606963688' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2317307583"}]': finished 2024-04-23T03:31:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:16 smithi149 ceph-mon[22716]: osdmap e100: 8 total, 8 up, 8 in 2024-04-23T03:31:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:16 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1848217604' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]: dispatch 2024-04-23T03:31:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:16 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]: dispatch 2024-04-23T03:31:16.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-mgr[57382]: -- 172.21.15.7:0/3260284306 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56123644a5a0 con 0x56123641e800 2024-04-23T03:31:16.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 bash[57341]: 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 2024-04-23T03:31:16.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 systemd[1]: Started Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:16.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:16.524+0000 7f383ade0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:31:16.727 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:16.725+0000 7f383ade0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:31:17.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:16.835+0000 7f383ade0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:31:17.562 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:17.207+0000 7f383ade0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:31:17.562 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:17.300+0000 7f383ade0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]': finished 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: osdmap e101: 8 total, 8 up, 8 in 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/4206721067' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]: dispatch 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]: dispatch 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-23T03:31:17.563 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]': finished 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: osdmap e101: 8 total, 8 up, 8 in 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/4206721067' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]: dispatch 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]: dispatch 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:17 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3858595978"}]': finished 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: osdmap e101: 8 total, 8 up, 8 in 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4206721067' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]: dispatch 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]: dispatch 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:17 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:18.807 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:18 smithi149 ceph-mon[22716]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]': finished 2024-04-23T03:31:18.807 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:18 smithi149 ceph-mon[22716]: osdmap e102: 8 total, 8 up, 8 in 2024-04-23T03:31:18.807 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:18 smithi149 ceph-mon[22716]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-23T03:31:18.807 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:18 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1163208710' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3848738408"}]: dispatch 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[26683]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]': finished 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[26683]: osdmap e102: 8 total, 8 up, 8 in 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[26683]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1163208710' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3848738408"}]: dispatch 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:18.624+0000 7f383ade0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:18.827+0000 7f383ade0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[21886]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/3848738408"}]': finished 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[21886]: osdmap e102: 8 total, 8 up, 8 in 2024-04-23T03:31:18.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[21886]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-04-23T03:31:18.830 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:18 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1163208710' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3848738408"}]: dispatch 2024-04-23T03:31:19.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:19.032+0000 7f383ade0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:31:19.659 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1163208710' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3848738408"}]': finished 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-mon[26683]: osdmap e103: 8 total, 8 up, 8 in 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1163208710' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3848738408"}]': finished 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-mon[21886]: osdmap e103: 8 total, 8 up, 8 in 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:19.341+0000 7f383ade0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:31:19.660 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:19.442+0000 7f383ade0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:31:19.736 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:19 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1163208710' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/3848738408"}]': finished 2024-04-23T03:31:19.736 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:19 smithi149 ceph-mon[22716]: osdmap e103: 8 total, 8 up, 8 in 2024-04-23T03:31:19.736 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:19 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:20.076 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:19.658+0000 7f383ade0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:31:20.076 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:19 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:19.877+0000 7f383ade0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:31:20.171 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:19 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:31:19] "GET /metrics HTTP/1.1" 200 38145 "" "Prometheus/2.33.4" 2024-04-23T03:31:20.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:20 smithi007 ceph-mon[26683]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:31:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:20 smithi007 ceph-mon[21886]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:31:20.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:20 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:20.515+0000 7f383ade0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:31:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:20 smithi149 ceph-mon[22716]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:31:21.324 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:21.000+0000 7f383ade0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:31:21.324 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:21.107+0000 7f383ade0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:31:21.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:21.323+0000 7f383ade0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T03:31:21.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-mon[26683]: from='client.34985 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:31:21.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:21.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-mon[21886]: from='client.34985 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:31:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:21.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:21 smithi149 ceph-mon[22716]: from='client.34985 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:31:21.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:21 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:21.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:21 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.279 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:21.903+0000 7f383ade0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:31:22.280 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:22.008+0000 7f383ade0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:31:22.280 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:22.278+0000 7f383ade0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:31:22.534 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:22.426+0000 7f383ade0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:31:22.534 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:22.533+0000 7f383ade0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:31:22.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:22.664+0000 7f383ade0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[26683]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[21886]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:22 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:22 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:22 smithi149 ceph-mon[22716]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:22 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:22 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:23.284 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:22.994+0000 7f383ade0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:31:23.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:23 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:24.000 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:23.655+0000 7f383ade0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:31:24.000 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:23.891+0000 7f383ade0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:31:24.000 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:23 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:23] ENGINE Bus STARTING 2024-04-23T03:31:24.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: CherryPy Checker: 2024-04-23T03:31:24.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: The Application mounted at '' has an empty config. 2024-04-23T03:31:24.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:31:24.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:24] ENGINE Serving on http://:::9283 2024-04-23T03:31:24.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:24] ENGINE Bus STARTED 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 670 B/s rd, 0 op/s 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: Standby manager daemon y restarted 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: Standby manager daemon y started 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T03:31:25.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:24 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:25.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 670 B/s rd, 0 op/s 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: Standby manager daemon y restarted 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: Standby manager daemon y started 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 670 B/s rd, 0 op/s 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: Standby manager daemon y restarted 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: Standby manager daemon y started 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-23T03:31:25.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:25.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-23T03:31:25.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:24 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:25.842 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:31:26.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:25 smithi007 ceph-mon[26683]: mgrmap e27: x(active, since 52s), standbys: y 2024-04-23T03:31:26.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:25 smithi007 ceph-mon[21886]: mgrmap e27: x(active, since 52s), standbys: y 2024-04-23T03:31:26.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:25 smithi149 ceph-mon[22716]: mgrmap e27: x(active, since 52s), standbys: y 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (30s) 5s ago 5m 15.3M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (16s) 4s ago 5m 40.9M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (16s) 5s ago 4m 76.9M - 3.5 e1d6a67b021e 4ce78e631caf 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283 running (71s) 4s ago 9m 509M - 19.0.0-3198-g56e81df3 82086b19bb07 ff28fa5468e0 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (10s) 5s ago 11m 65.7M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (11m) 5s ago 11m 56.4M 2048M 17.2.0 e1d6a67b021e 06ec9e9186b9 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (10m) 4s ago 10m 40.9M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (10m) 5s ago 10m 42.7M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (35s) 5s ago 6m 8594k - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (20s) 4s ago 6m 8627k - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:31:26.534 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (9m) 5s ago 9m 54.8M 2737M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (9m) 5s ago 9m 55.4M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (8m) 5s ago 8m 48.1M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (8m) 5s ago 8m 52.3M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (7m) 4s ago 7m 55.3M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (7m) 4s ago 7m 50.6M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (7m) 4s ago 7m 51.2M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (6m) 4s ago 6m 55.7M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (22s) 4s ago 6m 48.0M - 2.33.4 514e6a882f6e 3ef490c5c22e 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (4m) 5s ago 4m 72.1M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (4m) 4s ago 4m 74.5M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (4m) 5s ago 4m 73.5M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:31:26.535 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (4m) 4s ago 4m 73.4M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:26.969 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:31:26.970 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:31:27.322 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "2/23 daemons upgraded", 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mgr daemons", 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:31:27.323 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:31:27.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1877319129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:31:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:27.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:27 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1877319129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='client.15306 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='client.15312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:31:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:27.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:27 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1877319129' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:31:27.739 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:31:28.547 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[21886]: Reconfiguring iscsi.foo.smithi007.xwmqjz (dependencies changed)... 2024-04-23T03:31:28.547 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[21886]: Reconfiguring daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:31:28.547 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[21886]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:28.547 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/1226762494' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:31:28.548 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[26683]: Reconfiguring iscsi.foo.smithi007.xwmqjz (dependencies changed)... 2024-04-23T03:31:28.548 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[26683]: Reconfiguring daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:31:28.548 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[26683]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:28.548 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:28 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/1226762494' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:31:28.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:28 smithi149 ceph-mon[22716]: Reconfiguring iscsi.foo.smithi007.xwmqjz (dependencies changed)... 2024-04-23T03:31:28.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:28 smithi149 ceph-mon[22716]: Reconfiguring daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:31:28.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:28 smithi149 ceph-mon[22716]: from='client.15318 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:28.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:28 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/1226762494' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:31:28.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:29.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:29 smithi007 ceph-mon[26683]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:29.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:29 smithi007 ceph-mon[26683]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:31:29.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:29 smithi007 ceph-mon[21886]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:29.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:29 smithi007 ceph-mon[21886]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:31:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-mon[22716]: from='client.15330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-mon[22716]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:31:30.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:31:29] "GET /metrics HTTP/1.1" 200 38145 "" "Prometheus/2.33.4" 2024-04-23T03:31:30.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:29.898+0000 7f660e5a0700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Creating ceph-iscsi config... 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Traceback (most recent call last): 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.173 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: exec(code, run_globals) 2024-04-23T03:31:30.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.174 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Traceback (most recent call last): 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:30.175 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return future.result(timeout) 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return self.__get_result() 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: raise self._exception 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: use_current_daemon_image=reconfig, 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.176 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Creating ceph-iscsi config... 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: Traceback (most recent call last): 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: exec(code, run_globals) 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.177 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.178 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:29 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.583 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.584 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:30.585 INFO:journalctl@ceph.mon.a.smithi007.stdout: return future.result(timeout) 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: return self.__get_result() 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: raise self._exception 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_current_daemon_image=reconfig, 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.586 INFO:journalctl@ceph.mon.a.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:30.587 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.588 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.588 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:30] ENGINE Bus STOPPING 2024-04-23T03:31:30.588 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:31:30.588 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:30] ENGINE Bus STOPPED 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:30] ENGINE Bus STARTING 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:30] ENGINE Serving on http://:::9283 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:30] ENGINE Bus STARTED 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mon.c.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:30.589 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:30.590 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.591 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: return future.result(timeout) 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: return self.__get_result() 2024-04-23T03:31:30.592 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: raise self._exception 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_current_daemon_image=reconfig, 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.593 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.594 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.a.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.a.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.a.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: Failing over to other MGR 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[21886]: osdmap e104: 8 total, 8 up, 8 in 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.596 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: Failing over to other MGR 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-23T03:31:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:30 smithi007 ceph-mon[26683]: osdmap e104: 8 total, 8 up, 8 in 2024-04-23T03:31:30.609 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:30.243+0000 7f667c00a700 -1 mgr handle_mgr_map I was active but no longer am 2024-04-23T03:31:30.609 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ignoring --setuser ceph since I am not root 2024-04-23T03:31:30.609 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ignoring --setgroup ceph since I am not root 2024-04-23T03:31:30.609 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mgr[43963]: -- 172.21.15.149:0/1489593032 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd588e65a0 con 0x55cd588ba800 2024-04-23T03:31:30.609 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:30.417+0000 7fec53bb8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:31:30.610 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:30.607+0000 7fec53bb8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:31:30.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:31:30.610 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.610 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:30.610 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.611 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-04-23T03:31:30.612 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.event_loop.get_result(coro, timeout) 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: return future.result(timeout) 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: return self.__get_result() 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: raise self._exception 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-04-23T03:31:30.613 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_current_daemon_image=reconfig, 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: Creating ceph-iscsi config... 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: Non-zero exit code 1 from systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.614 INFO:journalctl@ceph.mon.b.smithi149.stdout: Traceback (most recent call last): 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: exec(code, run_globals) 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:31:30.615 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1123, in deploy_daemon 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: RuntimeError: Failed command: systemctl restart ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: Upgrade: Need to upgrade myself (mgr.x) 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: Failing over to other MGR 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: from='mgr.34886 172.21.15.149:0/2093416127' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-04-23T03:31:30.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-mon[22716]: osdmap e104: 8 total, 8 up, 8 in 2024-04-23T03:31:30.921 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:30.715+0000 7fec53bb8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:31:31.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:31.046+0000 7fec53bb8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:31:31.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:31.146+0000 7fec53bb8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:31:31.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.34886 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-23T03:31:31.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: mgrmap e28: y(active, starting, since 0.331453s) 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:31:31.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:31:31.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:31:31.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: Manager daemon y is now available 2024-04-23T03:31:31.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:31.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:31:31.424 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:31 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:31:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.34886 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-23T03:31:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: mgrmap e28: y(active, starting, since 0.331453s) 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:31:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: Manager daemon y is now available 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.34886 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: mgrmap e28: y(active, starting, since 0.331453s) 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: Standby manager daemon x started 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:31:31.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: Manager daemon y is now available 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:31:31.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:31 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:31:32.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: [23/Apr/2024:03:31:30] ENGINE Bus STARTING 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: [23/Apr/2024:03:31:31] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: [23/Apr/2024:03:31:31] ENGINE Serving on http://172.21.15.7:8765 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: [23/Apr/2024:03:31:31] ENGINE Bus STARTED 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: mgrmap e29: y(active, since 1.34175s), standbys: x 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[26683]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: [23/Apr/2024:03:31:30] ENGINE Bus STARTING 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: [23/Apr/2024:03:31:31] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: [23/Apr/2024:03:31:31] ENGINE Serving on http://172.21.15.7:8765 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: [23/Apr/2024:03:31:31] ENGINE Bus STARTED 2024-04-23T03:31:32.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: mgrmap e29: y(active, since 1.34175s), standbys: x 2024-04-23T03:31:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:31:32.570 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:32 smithi007 ceph-mon[21886]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:32.587 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:32.412+0000 7fec53bb8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: [23/Apr/2024:03:31:30] ENGINE Bus STARTING 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: [23/Apr/2024:03:31:31] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: [23/Apr/2024:03:31:31] ENGINE Serving on http://172.21.15.7:8765 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: [23/Apr/2024:03:31:31] ENGINE Bus STARTED 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: mgrmap e29: y(active, since 1.34175s), standbys: x 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:31:32.588 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-mon[22716]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:32.921 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:32.586+0000 7fec53bb8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:31:32.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:32 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:32.783+0000 7fec53bb8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:31:33.416 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-mon[22716]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:33.416 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-mon[22716]: mgrmap e30: y(active, since 2s), standbys: x 2024-04-23T03:31:33.416 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:33.098+0000 7fec53bb8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:31:33.416 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:33.199+0000 7fec53bb8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:31:33.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:33 smithi007 ceph-mon[26683]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:33.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:33 smithi007 ceph-mon[26683]: mgrmap e30: y(active, since 2s), standbys: x 2024-04-23T03:31:33.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:33 smithi007 ceph-mon[21886]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:33.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:33 smithi007 ceph-mon[21886]: mgrmap e30: y(active, since 2s), standbys: x 2024-04-23T03:31:33.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:33.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:33.415+0000 7fec53bb8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:31:33.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:33.632+0000 7fec53bb8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:31:34.629 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:34 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:34.273+0000 7fec53bb8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:31:34.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:34 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:34.721+0000 7fec53bb8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:31:34.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:34 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:34.821+0000 7fec53bb8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:31:35.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:35.044+0000 7fec53bb8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T03:31:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-mon[22716]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-mon[22716]: mgrmap e31: y(active, since 4s), standbys: x 2024-04-23T03:31:35.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[26683]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[26683]: mgrmap e31: y(active, since 4s), standbys: x 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[21886]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[21886]: mgrmap e31: y(active, since 4s), standbys: x 2024-04-23T03:31:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:35 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:35.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:35.624+0000 7fec53bb8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:31:35.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:35.724+0000 7fec53bb8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:31:36.361 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:35 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:35.991+0000 7fec53bb8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:31:36.361 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:36.137+0000 7fec53bb8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:31:36.361 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:36.240+0000 7fec53bb8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:31:36.362 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:36.360+0000 7fec53bb8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:31:36.921 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:36.670+0000 7fec53bb8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:31:37.286 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:36 smithi149 ceph-mon[22716]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:31:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[21886]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:31:37.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:36 smithi007 ceph-mon[26683]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:37.284+0000 7fec53bb8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:37.507+0000 7fec53bb8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:37] ENGINE Bus STARTING 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: CherryPy Checker: 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: The Application mounted at '' has an empty config. 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:37] ENGINE Serving on http://:::9283 2024-04-23T03:31:37.614 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:37 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: [23/Apr/2024:03:31:37] ENGINE Bus STARTED 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[21886]: Standby manager daemon x restarted 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[21886]: Standby manager daemon x started 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[26683]: Standby manager daemon x restarted 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:31:38.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:31:38.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:38.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:31:38.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:38 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:38.659 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-mon[22716]: Standby manager daemon x restarted 2024-04-23T03:31:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:31:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:31:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:31:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/2851392754' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:39.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:39 smithi007 ceph-mon[26683]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-23T03:31:39.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:39 smithi007 ceph-mon[26683]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-23T03:31:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:39 smithi007 ceph-mon[21886]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-23T03:31:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:39 smithi007 ceph-mon[21886]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-23T03:31:39.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:39 smithi149 ceph-mon[22716]: mgrmap e32: y(active, since 8s), standbys: x 2024-04-23T03:31:39.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:39 smithi149 ceph-mon[22716]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 29 KiB/s rd, 0 B/s wr, 16 op/s 2024-04-23T03:31:40.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:39 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[43959]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:31:39] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-23T03:31:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:31:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:31:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[26683]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:40.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:31:40.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:31:40.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:40 smithi007 ceph-mon[21886]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-23T03:31:40.586 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:40.586 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:40.586 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:40.586 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:31:40.586 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:31:40.587 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:40 smithi149 ceph-mon[22716]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-23T03:31:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[26683]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:31:41.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:41 smithi007 ceph-mon[21886]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-23T03:31:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:41 smithi149 ceph-mon[22716]: Reconfiguring daemon prometheus.a on smithi149 2024-04-23T03:31:42.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:42 smithi149 ceph-mon[22716]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:31:42.768 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:42 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3200836804' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:31:42.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:42 smithi007 ceph-mon[26683]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:31:42.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:42 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3200836804' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:31:42.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:42 smithi007 ceph-mon[21886]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:31:42.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:42 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3200836804' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:31:43.749 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:43 smithi149 ceph-mon[22716]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:43.750 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:43.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:43 smithi007 ceph-mon[21886]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:43.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:43 smithi007 ceph-mon[26683]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:44.045 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:43 smithi149 systemd[1]: Stopping Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:44.045 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:43 smithi149 bash[50621]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.044Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-23T03:31:44.336 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[47429]: ts=2024-04-23T03:31:44.045Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 podman[50634]: 2024-04-23 03:31:44.055269719 +0000 UTC m=+0.127829060 container died 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 podman[50634]: 2024-04-23 03:31:44.335977526 +0000 UTC m=+0.408536876 container cleanup 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:44.337 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 bash[50634]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a 2024-04-23T03:31:44.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 bash[50660]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-23T03:31:44.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 podman[50648]: 2024-04-23 03:31:44.553762407 +0000 UTC m=+0.496617810 container remove 3ef490c5c22e07ee192899afb4774bcb621e206c090cab7078e198e1e9766041 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:44.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-23T03:31:44.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 systemd[1]: Stopped Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:44.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 systemd[1]: Starting Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:45.055 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 podman[50729]: 2024-04-23 03:31:44.68773117 +0000 UTC m=+0.018821414 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-23T03:31:45.055 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:44 smithi149 podman[50729]: 2024-04-23 03:31:44.82971645 +0000 UTC m=+0.160806679 container create 86ffd466d290094f1166865f0a5315bd3ac67ab1fe4bca00d5d73dccd43e45fc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 podman[50729]: 2024-04-23 03:31:45.157202989 +0000 UTC m=+0.488293215 container init 86ffd466d290094f1166865f0a5315bd3ac67ab1fe4bca00d5d73dccd43e45fc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 podman[50729]: 2024-04-23 03:31:45.159710552 +0000 UTC m=+0.490800778 container start 86ffd466d290094f1166865f0a5315bd3ac67ab1fe4bca00d5d73dccd43e45fc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.184Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.184Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.184Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.184Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi149 (none))" 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.184Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.184Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-23T03:31:45.393 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.186Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.187Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.188Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.190Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.190Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.825µs 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.190Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 bash[50729]: 86ffd466d290094f1166865f0a5315bd3ac67ab1fe4bca00d5d73dccd43e45fc 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 systemd[1]: Started Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:45.394 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.392Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-04-23T03:31:45.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:45 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:45] ENGINE Bus STOPPING 2024-04-23T03:31:45.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:45 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:31:45.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:45 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:45] ENGINE Bus STOPPED 2024-04-23T03:31:45.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:45 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:45] ENGINE Bus STARTING 2024-04-23T03:31:45.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:45 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:45] ENGINE Serving on http://:::9283 2024-04-23T03:31:45.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:45 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:31:45] ENGINE Bus STARTED 2024-04-23T03:31:45.671 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.626Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.825Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.826Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.826Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=35.05µs wal_replay_duration=635.25205ms total_replay_duration=635.30174ms 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.831Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.831Z caller=main.go:947 level=info msg="TSDB started" 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.831Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.856Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=25.239933ms db_storage=1.074µs remote_storage=1.816µs web_handler=601ns query_engine=902ns scrape=649.583µs scrape_sd=180.082µs notify=24.969µs notify_sd=24.188µs rules=23.175391ms 2024-04-23T03:31:46.076 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:45.856Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-23T03:31:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:46.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:46.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-04-23T03:31:46.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-04-23T03:31:46.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:46.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:46 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:46.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:46.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-23T03:31:46.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-04-23T03:31:46.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi149.front.sepia.ceph.com:9095"}]: dispatch 2024-04-23T03:31:46.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:31:46.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:46.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:46 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:31:47.523 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:47 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:47.523 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:47 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:31:47.523 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:47 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:47 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:47 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:31:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:47 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:47.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:47 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:31:47.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:47 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:31:47.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:47 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:31:48.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:48 smithi149 ceph-mon[22716]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:48.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:48 smithi149 ceph-mon[22716]: Upgrade: Updating mgr.x 2024-04-23T03:31:48.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:48 smithi149 ceph-mon[22716]: Deploying daemon mgr.x on smithi149 2024-04-23T03:31:48.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:48 smithi007 ceph-mon[26683]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:48.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:48 smithi007 ceph-mon[26683]: Upgrade: Updating mgr.x 2024-04-23T03:31:48.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:48 smithi007 ceph-mon[26683]: Deploying daemon mgr.x on smithi149 2024-04-23T03:31:48.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:48 smithi007 ceph-mon[21886]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:31:48.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:48 smithi007 ceph-mon[21886]: Upgrade: Updating mgr.x 2024-04-23T03:31:48.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:48 smithi007 ceph-mon[21886]: Deploying daemon mgr.x on smithi149 2024-04-23T03:31:48.907 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:48 smithi149 systemd[1]: Stopping Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:49.171 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:48 smithi149 podman[51070]: 2024-04-23 03:31:48.906341053 +0000 UTC m=+0.166999229 container died ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-56e81df, maintainer=Guillaume Abrioux ) 2024-04-23T03:31:49.633 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 podman[51070]: 2024-04-23 03:31:49.284623781 +0000 UTC m=+0.545281997 container cleanup ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, RELEASE=main-56e81df, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T03:31:49.633 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 bash[51070]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x 2024-04-23T03:31:49.633 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 podman[51085]: 2024-04-23 03:31:49.519342769 +0000 UTC m=+0.616504255 container remove ff28fa5468e020a9a0c19893f2c480a2fbd2b89f9e6f22f033bfa405fda02e71 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-56e81df, org.label-schema.license=GPLv2) 2024-04-23T03:31:49.633 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:31:49.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-04-23T03:31:49.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 systemd[1]: Stopped Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:49.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service: Consumed 26.689s CPU time. 2024-04-23T03:31:49.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:49 smithi149 systemd[1]: Starting Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:31:50.390 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:50 smithi149 ceph-mon[22716]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 501 B/s rd, 0 op/s 2024-04-23T03:31:50.390 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 podman[51213]: 2024-04-23 03:31:49.976370392 +0000 UTC m=+0.017742255 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:31:50.391 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 podman[51213]: 2024-04-23 03:31:50.170775795 +0000 UTC m=+0.212147631 container create d525caf93187836cd5751d67498e5a1410f96113d3c5ae0bae251fdd6ff2fb61 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T03:31:50.391 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 podman[51213]: 2024-04-23 03:31:50.389823997 +0000 UTC m=+0.431195842 container init d525caf93187836cd5751d67498e5a1410f96113d3c5ae0bae251fdd6ff2fb61 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T03:31:50.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:50 smithi007 ceph-mon[26683]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 501 B/s rd, 0 op/s 2024-04-23T03:31:50.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:50 smithi007 ceph-mon[21886]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 501 B/s rd, 0 op/s 2024-04-23T03:31:50.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 podman[51213]: 2024-04-23 03:31:50.39247757 +0000 UTC m=+0.433849412 container start d525caf93187836cd5751d67498e5a1410f96113d3c5ae0bae251fdd6ff2fb61 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:31:50.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 ceph-mgr[51233]: -- 172.21.15.149:0/2497410110 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563462f445a0 con 0x563462f18c00 2024-04-23T03:31:50.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 bash[51213]: d525caf93187836cd5751d67498e5a1410f96113d3c5ae0bae251fdd6ff2fb61 2024-04-23T03:31:50.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 systemd[1]: Started Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:31:50.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:50.541+0000 7f152230e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:31:51.171 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:50.737+0000 7f152230e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:31:51.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:50.849+0000 7f152230e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:31:51.456 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:51.192+0000 7f152230e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:31:51.456 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:51.287+0000 7f152230e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:31:51.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:51 smithi007 ceph-mon[21886]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 501 B/s rd, 0 op/s 2024-04-23T03:31:51.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:51 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:51.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:51 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:51.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:51 smithi007 ceph-mon[26683]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 501 B/s rd, 0 op/s 2024-04-23T03:31:51.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:51 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:51.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:51 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:51.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:51 smithi149 ceph-mon[22716]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 501 B/s rd, 0 op/s 2024-04-23T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:51 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:51.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:51 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:52.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:52 smithi149 ceph-mon[22716]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:31:52.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:52.539+0000 7f152230e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:31:53.042 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:52.749+0000 7f152230e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:31:53.042 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:52.935+0000 7f152230e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:31:53.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:52 smithi007 ceph-mon[26683]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:31:53.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:52 smithi007 ceph-mon[21886]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:31:53.336 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:53.236+0000 7f152230e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:31:53.336 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:53.335+0000 7f152230e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:31:53.599 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:53.599 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:53.564+0000 7f152230e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:31:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:53 smithi149 ceph-mon[22716]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:53.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:53.764+0000 7f152230e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:31:54.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:53 smithi007 ceph-mon[26683]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:54.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:53 smithi007 ceph-mon[21886]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:54.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:54.392+0000 7f152230e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:31:55.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:54.838+0000 7f152230e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:31:55.172 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:54.935+0000 7f152230e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:31:55.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:55.179+0000 7f152230e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T03:31:56.109 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:55.736+0000 7f152230e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:31:56.110 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:55.837+0000 7f152230e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:31:56.387 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-mon[22716]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:56.388 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:56.388 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:56.388 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:56.115+0000 7f152230e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:31:56.388 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:56.280+0000 7f152230e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:31:56.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:31:56] "GET /metrics HTTP/1.1" 200 38232 "" "Prometheus/2.33.4" 2024-04-23T03:31:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-mon[21886]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-mon[26683]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:31:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:56 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:56.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:56.386+0000 7f152230e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:31:56.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:56.511+0000 7f152230e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:31:57.149 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:56.849+0000 7f152230e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:31:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:57.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:57 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:57.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:57 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:57 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:57.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:57 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:31:57.770 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:57.484+0000 7f152230e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:31:57.770 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:57.713+0000 7f152230e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:31:57.771 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: [23/Apr/2024:03:31:57] ENGINE Bus STARTING 2024-04-23T03:31:58.097 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: CherryPy Checker: 2024-04-23T03:31:58.098 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: The Application mounted at '' has an empty config. 2024-04-23T03:31:58.098 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:31:58.098 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: [23/Apr/2024:03:31:57] ENGINE Serving on http://:::9283 2024-04-23T03:31:58.098 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:31:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: [23/Apr/2024:03:31:57] ENGINE Bus STARTED 2024-04-23T03:31:58.117 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: Standby manager daemon x restarted 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:31:58.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:58.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:58.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: Standby manager daemon x restarted 2024-04-23T03:31:58.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: Standby manager daemon x started 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[21886]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: Standby manager daemon x restarted 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:31:58.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:58 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3190309316' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:31:58.800 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:31:58.800 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (63s) 23s ago 5m 15.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:31:58.800 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (48s) 3s ago 5m 45.5M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 unknown 23s ago 4m - - 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (8s) 3s ago 10m 68.5M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (42s) 23s ago 12m 475M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (12m) 23s ago 12m 57.2M 2048M 17.2.0 e1d6a67b021e 06ec9e9186b9 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (10m) 3s ago 10m 46.4M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (10m) 23s ago 10m 43.8M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (68s) 23s ago 6m 8732k - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (52s) 3s ago 6m 8358k - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (10m) 23s ago 10m 55.0M 2737M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (9m) 23s ago 9m 55.7M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (9m) 23s ago 9m 48.4M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (8m) 23s ago 8m 52.6M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (8m) 3s ago 8m 55.9M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (8m) 3s ago 8m 51.1M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:31:58.801 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (7m) 3s ago 7m 52.0M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:31:58.802 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (7m) 3s ago 7m 55.2M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:31:58.802 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (14s) 3s ago 6m 46.2M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:31:58.802 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (5m) 23s ago 5m 72.6M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:31:58.802 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (5m) 3s ago 5m 75.2M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:31:58.802 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (5m) 23s ago 4m 73.9M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:31:58.802 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (4m) 3s ago 4m 74.4M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:31:58.911 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:31:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:31:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:31:59.218 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:31:59.218 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-23T03:31:59.219 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:31:59.220 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:31:59.537 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:59 smithi149 ceph-mon[22716]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:59.537 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:59 smithi149 ceph-mon[22716]: mgrmap e33: y(active, since 28s), standbys: x 2024-04-23T03:31:59.537 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:31:59 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/3237918450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:31:59.567 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:31:59.568 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "2/23 daemons upgraded", 2024-04-23T03:31:59.568 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mgr daemons", 2024-04-23T03:31:59.568 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:31:59.568 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:31:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:59 smithi007 ceph-mon[26683]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:59 smithi007 ceph-mon[26683]: mgrmap e33: y(active, since 28s), standbys: x 2024-04-23T03:31:59.568 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:31:59 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/3237918450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:31:59.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:59 smithi007 ceph-mon[21886]: from='client.25312 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:31:59.568 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:59 smithi007 ceph-mon[21886]: mgrmap e33: y(active, since 28s), standbys: x 2024-04-23T03:31:59.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:31:59 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/3237918450' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:31:59.997 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-23T03:31:59.997 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:31:59.997 INFO:teuthology.orchestra.run.smithi007.stdout: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:32:00.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:00.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:00.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-23T03:32:00.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[26683]: from='client.? 172.21.15.7:0/4268793773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:00.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-23T03:32:00.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-23T03:32:00.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-23T03:32:00.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:00.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-23T03:32:00.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:00 smithi007 ceph-mon[21886]: from='client.? 172.21.15.7:0/4268793773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:32:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='client.25318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='client.25324 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-04-23T03:32:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-04-23T03:32:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-04-23T03:32:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-04-23T03:32:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-04-23T03:32:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:00 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4268793773' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:32:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[26683]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[26683]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-04-23T03:32:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[26683]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[26683]: Upgrade: Setting container_image for all mgr 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[26683]: Upgrade: It appears safe to stop mon.a 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[21886]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[21886]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[21886]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[21886]: Upgrade: Setting container_image for all mgr 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[21886]: Upgrade: It appears safe to stop mon.a 2024-04-23T03:32:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:01 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:01.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:01 smithi149 ceph-mon[22716]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:01 smithi149 ceph-mon[22716]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 920 B/s rd, 0 op/s 2024-04-23T03:32:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:01 smithi149 ceph-mon[22716]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:32:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:01 smithi149 ceph-mon[22716]: Upgrade: Setting container_image for all mgr 2024-04-23T03:32:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:01 smithi149 ceph-mon[22716]: Upgrade: It appears safe to stop mon.a 2024-04-23T03:32:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:01 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: Upgrade: Updating mon.a 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: Deploying daemon mon.a on smithi007 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[26683]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:02.778 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: Upgrade: Updating mon.a 2024-04-23T03:32:02.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:02.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:02.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:02.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:02.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: Deploying daemon mon.a on smithi007 2024-04-23T03:32:02.779 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:02 smithi007 ceph-mon[21886]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: Upgrade: Updating mon.a 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: Deploying daemon mon.a on smithi007 2024-04-23T03:32:02.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:02 smithi149 ceph-mon[22716]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:32:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:03 smithi007 systemd[1]: Stopping Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:32:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a[21882]: 2024-04-23T03:32:03.682+0000 7ff664bd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:32:03.696 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a[21882]: 2024-04-23T03:32:03.682+0000 7ff664bd4700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-23T03:32:03.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:04.001 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:03 smithi007 podman[61598]: 2024-04-23 03:32:03.754404064 +0000 UTC m=+0.171942488 container died 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, RELEASE=HEAD, name=centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-04-23T03:32:04.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:03] ENGINE Bus STOPPING 2024-04-23T03:32:04.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:32:04.001 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:03] ENGINE Bus STOPPED 2024-04-23T03:32:04.002 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:03] ENGINE Bus STARTING 2024-04-23T03:32:04.002 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:03] ENGINE Serving on http://:::9283 2024-04-23T03:32:04.002 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:03 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:03] ENGINE Bus STARTED 2024-04-23T03:32:04.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 podman[61598]: 2024-04-23 03:32:04.116618182 +0000 UTC m=+0.534156611 container cleanup 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_CLEAN=True, release=754, version=8, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, ceph=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-04-23T03:32:04.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 bash[61598]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a 2024-04-23T03:32:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 podman[61623]: 2024-04-23 03:32:04.317739343 +0000 UTC m=+0.561231518 container remove 06ec9e9186b9d3bc6cf138ef0011c4ca81d991d2ae47c89c63ad0be75ebf1ab9 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, architecture=x86_64, com.redhat.component=centos-stream-container, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870) 2024-04-23T03:32:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-23T03:32:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 systemd[1]: Stopped Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:32:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service: Consumed 15.633s CPU time. 2024-04-23T03:32:05.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 systemd[1]: Starting Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:32:05.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 podman[61753]: 2024-04-23 03:32:04.960983841 +0000 UTC m=+0.106941879 container create 72e8f30ba3f4d10d4826f7d2a11c2f3f9bad187825c9a0b845f0d61992969d1d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:32:05.077 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:04 smithi007 podman[61753]: 2024-04-23 03:32:04.872456318 +0000 UTC m=+0.018414385 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:32:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 podman[61753]: 2024-04-23 03:32:05.16393057 +0000 UTC m=+0.309888628 container init 72e8f30ba3f4d10d4826f7d2a11c2f3f9bad187825c9a0b845f0d61992969d1d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-23T03:32:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 podman[61753]: 2024-04-23 03:32:05.167458325 +0000 UTC m=+0.313416365 container start 72e8f30ba3f4d10d4826f7d2a11c2f3f9bad187825c9a0b845f0d61992969d1d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:32:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T03:32:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev), process ceph-mon, pid 2 2024-04-23T03:32:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: pidfile_write: ignore empty --pid-file 2024-04-23T03:32:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: load: jerasure load: lrc 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: RocksDB version: 7.9.2 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Git sha 0 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Compile date 2024-04-22 15:21:46 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: DB SUMMARY 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: DB Session ID: EHGPIF4FDDLP0ZO6JV48 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: CURRENT file: CURRENT 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: MANIFEST file: MANIFEST-000015 size: 980 Bytes 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3693613 ; 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.error_if_exists: 0 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.create_if_missing: 0 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.paranoid_checks: 1 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T03:32:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.env: 0x5585ec9d9cc0 2024-04-23T03:32:05.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T03:32:05.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.info_log: 0x5585ed4c7be0 2024-04-23T03:32:05.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T03:32:05.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.statistics: (nil) 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.use_fsync: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_log_file_size: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.allow_fallocate: 1 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.use_direct_reads: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T03:32:05.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.db_log_dir: 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.wal_dir: 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.write_buffer_manager: 0x5585ed45b0e0 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T03:32:05.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.unordered_write: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.row_cache: None 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.wal_filter: None 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.two_write_queues: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.wal_compression: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.atomic_flush: 0 2024-04-23T03:32:05.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.log_readahead_size: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_background_jobs: 2 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_background_compactions: -1 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_subcompactions: 1 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T03:32:05.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_open_files: -1 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_background_flushes: -1 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Compression algorithms supported: 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kZSTD supported: 0 2024-04-23T03:32:05.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kXpressCompression supported: 0 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kZlibCompression supported: 1 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kSnappyCompression supported: 1 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kLZ4Compression supported: 1 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: kBZip2Compression supported: 0 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.merge_operator: 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_filter: None 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T03:32:05.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5585ed4c7dc0) 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 4 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x5585ed48d1f0 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-04-23T03:32:05.592 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: bloomfilter 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-23T03:32:05.593 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 5 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout: max_auto_readahead_size: 262144 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout: prepopulate_block_cache: 0 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout: initial_auto_readahead_size: 8192 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression: NoCompression 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.num_levels: 7 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T03:32:05.594 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T03:32:05.595 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T03:32:05.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T03:32:05.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.inplace_update_support: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.bloom_locality: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.max_successive_merges: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.ttl: 2592000 2024-04-23T03:32:05.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.enable_blob_files: false 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.min_blob_size: 0 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T03:32:05.599 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12075, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c0d8d8fb-14f2-46ed-9337-9ccff6a2b7e0 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843125197558, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843125208243, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3427510, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12076, "largest_seqno": 13487, "table_properties": {"data_size": 3420426, "index_size": 4439, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 15498, "raw_average_key_size": 24, "raw_value_size": 3406257, "raw_average_value_size": 5322, "num_data_blocks": 201, "num_entries": 640, "num_filter_entries": 640, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713843125, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c0d8d8fb-14f2-46ed-9337-9ccff6a2b7e0", "db_session_id": "EHGPIF4FDDLP0ZO6JV48", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843125208356, "job": 1, "event": "recovery_finished"} 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-04-23T03:32:05.600 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5585ed5a2000 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: DB pointer 0x5585ed58c000 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T03:32:05.601 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 1/0 3.27 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: L6 1/0 7.68 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 10.95 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 611.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 611.3 0.01 0.00 1 0.005 0 0 0.0 0.0 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:32:05.602 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 247.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 247.05 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache BinnedLRUCache@0x5585ed48d1f0#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???) e3 preinit fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).mds e1 new map 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).mds e1 print_map 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T03:32:05.603 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).osd e104 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).osd e104 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 ceph-mon[61772]: mon.a@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 bash[61753]: 72e8f30ba3f4d10d4826f7d2a11c2f3f9bad187825c9a0b845f0d61992969d1d 2024-04-23T03:32:05.604 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:05 smithi007 systemd[1]: Started Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:32:06.506 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:32:06] "GET /metrics HTTP/1.1" 200 38229 "" "Prometheus/2.33.4" 2024-04-23T03:32:08.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ignoring --setuser ceph since I am not root 2024-04-23T03:32:08.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ignoring --setgroup ceph since I am not root 2024-04-23T03:32:08.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mgr[57382]: -- 172.21.15.7:0/1444080649 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a6b6ba25a0 con 0x55a6b6b76800 2024-04-23T03:32:08.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:08.428+0000 7f1cb9e41200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:32:08.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: mon.a calling monitor election 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: monmap epoch 3 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: last_changed 2024-04-23T03:21:21.198553+0000 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: min_mon_release 17 (quincy) 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: election_strategy: 1 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:08.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: fsmap 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: osdmap e104: 8 total, 8 up, 8 in 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: mgrmap e33: y(active, since 38s), standbys: x 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: from='mgr.25204 ' entity='' 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[26683]: mgrmap e34: y(active, since 38s), standbys: x 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: mon.a calling monitor election 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-04-23T03:32:08.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: monmap epoch 3 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: last_changed 2024-04-23T03:21:21.198553+0000 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: min_mon_release 17 (quincy) 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: election_strategy: 1 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: fsmap 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: osdmap e104: 8 total, 8 up, 8 in 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: mgrmap e33: y(active, since 38s), standbys: x 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:32:08.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:32:08.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: from='mgr.25204 ' entity='' 2024-04-23T03:32:08.587 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-mon[61772]: mgrmap e34: y(active, since 38s), standbys: x 2024-04-23T03:32:08.601 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: ignoring --setuser ceph since I am not root 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: ignoring --setgroup ceph since I am not root 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mgr[51233]: -- 172.21.15.149:0/3149691262 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e164f6a5a0 con 0x55e164f3ec00 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:08.415+0000 7fada829e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:08.599+0000 7fada829e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: from='mgr.25204 172.21.15.7:0/366965793' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: mon.a calling monitor election 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 561 B/s rd, 0 op/s 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: monmap epoch 3 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:08.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: last_changed 2024-04-23T03:21:21.198553+0000 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: min_mon_release 17 (quincy) 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: election_strategy: 1 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: fsmap 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: osdmap e104: 8 total, 8 up, 8 in 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: mgrmap e33: y(active, since 38s), standbys: x 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: from='mgr.25204 ' entity='' 2024-04-23T03:32:08.603 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-mon[22716]: mgrmap e34: y(active, since 38s), standbys: x 2024-04-23T03:32:08.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:08.703+0000 7fada829e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:32:08.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:09.053 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:08.622+0000 7f1cb9e41200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:32:09.053 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:08.730+0000 7f1cb9e41200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-23T03:32:09.053 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:09.052+0000 7f1cb9e41200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:32:09.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:09.140+0000 7f1cb9e41200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:32:09.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:09.012+0000 7fada829e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-23T03:32:09.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:09.098+0000 7fada829e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-23T03:32:10.567 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:10 smithi149 ceph-mon[22716]: mgrmap e35: y(active, since 39s), standbys: x 2024-04-23T03:32:10.567 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:10.290+0000 7fada829e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:32:10.567 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:10.468+0000 7fada829e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:32:10.569 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:10 smithi007 ceph-mon[26683]: mgrmap e35: y(active, since 39s), standbys: x 2024-04-23T03:32:10.569 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:10 smithi007 ceph-mon[61772]: mgrmap e35: y(active, since 39s), standbys: x 2024-04-23T03:32:10.570 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:10.390+0000 7f1cb9e41200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-23T03:32:10.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:10.568+0000 7f1cb9e41200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-23T03:32:10.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:10 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:10.747+0000 7f1cb9e41200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:32:10.922 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:10.652+0000 7fada829e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-23T03:32:11.278 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:10.964+0000 7fada829e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:32:11.278 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:11.062+0000 7fada829e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:32:11.278 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:11.276+0000 7fada829e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:32:11.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:11.052+0000 7f1cb9e41200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-23T03:32:11.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:11.148+0000 7f1cb9e41200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-23T03:32:11.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:11.479+0000 7fada829e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:32:11.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:11.361+0000 7f1cb9e41200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-23T03:32:11.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:11.564+0000 7f1cb9e41200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-23T03:32:12.421 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:12.088+0000 7fada829e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:32:12.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:12 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:12.193+0000 7f1cb9e41200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-23T03:32:12.824 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:12.517+0000 7fada829e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:32:12.824 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:12.616+0000 7fada829e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:32:12.824 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:12.822+0000 7fada829e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T03:32:12.958 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:12 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:12.640+0000 7f1cb9e41200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-23T03:32:12.958 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:12 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:12.743+0000 7f1cb9e41200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-23T03:32:13.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:12 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:12.957+0000 7f1cb9e41200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-23T03:32:13.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:13.363+0000 7fada829e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:32:13.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:13.458+0000 7fada829e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:32:13.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:13.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:13.512+0000 7f1cb9e41200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-23T03:32:13.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:13.612+0000 7f1cb9e41200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-23T03:32:14.071 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:13.714+0000 7fada829e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:32:14.071 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:13.848+0000 7fada829e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:32:14.072 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:13.951+0000 7fada829e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:32:14.072 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:14.070+0000 7fada829e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:32:14.240 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:13.871+0000 7f1cb9e41200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-23T03:32:14.240 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:14.013+0000 7f1cb9e41200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-23T03:32:14.241 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:14.115+0000 7f1cb9e41200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-23T03:32:14.241 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:14.239+0000 7f1cb9e41200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-23T03:32:14.672 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:14.379+0000 7fada829e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:32:14.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:14.555+0000 7f1cb9e41200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-23T03:32:15.278 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:14.989+0000 7fada829e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:32:15.278 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:15.213+0000 7fada829e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:32:15.278 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: [23/Apr/2024:03:32:15] ENGINE Bus STARTING 2024-04-23T03:32:15.532 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:15.198+0000 7f1cb9e41200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:15.435+0000 7f1cb9e41200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[26683]: Standby manager daemon x restarted 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[26683]: Standby manager daemon x started 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[26683]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:32:15.533 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[61772]: Standby manager daemon x restarted 2024-04-23T03:32:15.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[61772]: Standby manager daemon x started 2024-04-23T03:32:15.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[61772]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:32:15.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[61772]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:32:15.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[61772]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:32:15.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-mon[61772]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-mon[22716]: Standby manager daemon x restarted 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-mon[22716]: Standby manager daemon x started 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-mon[22716]: from='mgr.? 172.21.15.149:0/3851615515' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: CherryPy Checker: 2024-04-23T03:32:15.583 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: The Application mounted at '' has an empty config. 2024-04-23T03:32:15.584 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: 2024-04-23T03:32:15.584 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: [23/Apr/2024:03:32:15] ENGINE Serving on http://:::9283 2024-04-23T03:32:15.584 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:32:15 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x[51229]: [23/Apr/2024:03:32:15] ENGINE Bus STARTED 2024-04-23T03:32:15.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:15] ENGINE Bus STARTING 2024-04-23T03:32:15.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: CherryPy Checker: 2024-04-23T03:32:15.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: The Application mounted at '' has an empty config. 2024-04-23T03:32:15.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:15.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:15] ENGINE Serving on http://:::9283 2024-04-23T03:32:15.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:15 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:15] ENGINE Bus STARTED 2024-04-23T03:32:16.456 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:32:16] "GET /metrics HTTP/1.1" 200 35440 "" "Prometheus/2.33.4" 2024-04-23T03:32:16.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: mgrmap e36: y(active, since 45s), standbys: x 2024-04-23T03:32:16.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: Active manager daemon y restarted 2024-04-23T03:32:16.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: Activating manager daemon y 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: mgrmap e37: y(active, starting, since 0.00899404s), standbys: x 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:32:16.457 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: Manager daemon y is now available 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:32:16.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: mgrmap e36: y(active, since 45s), standbys: x 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: Active manager daemon y restarted 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: Activating manager daemon y 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: mgrmap e37: y(active, starting, since 0.00899404s), standbys: x 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:32:16.459 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: Manager daemon y is now available 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:32:16.460 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: mgrmap e36: y(active, since 45s), standbys: x 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: Active manager daemon y restarted 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: Activating manager daemon y 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: mgrmap e37: y(active, starting, since 0.00899404s), standbys: x 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-23T03:32:16.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: Manager daemon y is now available 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:16.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:32:16.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-23T03:32:16.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:32:16.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:16 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-23T03:32:16.820 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:16.457+0000 7f1c772de700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-04-23T03:32:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: [23/Apr/2024:03:32:16] ENGINE Bus STARTING 2024-04-23T03:32:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: mgrmap e38: y(active, since 1.00907s), standbys: x 2024-04-23T03:32:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: [23/Apr/2024:03:32:16] ENGINE Serving on http://172.21.15.7:8765 2024-04-23T03:32:17.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: [23/Apr/2024:03:32:16] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:32:17.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:17 smithi149 ceph-mon[22716]: [23/Apr/2024:03:32:16] ENGINE Bus STARTED 2024-04-23T03:32:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: [23/Apr/2024:03:32:16] ENGINE Bus STARTING 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: mgrmap e38: y(active, since 1.00907s), standbys: x 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: [23/Apr/2024:03:32:16] ENGINE Serving on http://172.21.15.7:8765 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: [23/Apr/2024:03:32:16] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[61772]: [23/Apr/2024:03:32:16] ENGINE Bus STARTED 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: [23/Apr/2024:03:32:16] ENGINE Bus STARTING 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: mgrmap e38: y(active, since 1.00907s), standbys: x 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: [23/Apr/2024:03:32:16] ENGINE Serving on http://172.21.15.7:8765 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: [23/Apr/2024:03:32:16] ENGINE Serving on https://172.21.15.7:7150 2024-04-23T03:32:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:17 smithi007 ceph-mon[26683]: [23/Apr/2024:03:32:16] ENGINE Bus STARTED 2024-04-23T03:32:18.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:18 smithi149 ceph-mon[22716]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-23T03:32:18.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:18 smithi149 ceph-mon[22716]: Cluster is now healthy 2024-04-23T03:32:18.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:18 smithi149 ceph-mon[22716]: mgrmap e39: y(active, since 2s), standbys: x 2024-04-23T03:32:18.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:18 smithi149 ceph-mon[22716]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:18.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[61772]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-23T03:32:18.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:32:18.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[61772]: mgrmap e39: y(active, since 2s), standbys: x 2024-04-23T03:32:18.783 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[61772]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:18.783 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[26683]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-23T03:32:18.783 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[26683]: Cluster is now healthy 2024-04-23T03:32:18.783 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[26683]: mgrmap e39: y(active, since 2s), standbys: x 2024-04-23T03:32:18.783 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:18 smithi007 ceph-mon[26683]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:18.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:20.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[61772]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[61772]: mgrmap e40: y(active, since 4s), standbys: x 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[26683]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[26683]: mgrmap e40: y(active, since 4s), standbys: x 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:20 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:20 smithi149 ceph-mon[22716]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:20 smithi149 ceph-mon[22716]: mgrmap e40: y(active, since 4s), standbys: x 2024-04-23T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:20 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:20 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:20 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:20.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:20 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:22 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:22 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:22 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:22 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:22 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:23.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:23 smithi149 ceph-mon[22716]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:23.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:23 smithi007 ceph-mon[61772]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:23.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:23 smithi007 ceph-mon[26683]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:32:23.909 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:23 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-23T03:32:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:25.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:25.586 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:25 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-23T03:32:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:25.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:25 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:26.386 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:32:26] "GET /metrics HTTP/1.1" 200 35440 "" "Prometheus/2.33.4" 2024-04-23T03:32:26.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:32:26.386 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.387 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:32:26.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.389 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:26.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:26.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:26.390 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:26 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Adjusting osd_memory_target on smithi149 to 174.0M 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Unable to set osd_memory_target on smithi149 to 182484172: error parsing value: Value '182484172' is below minimum 939524096 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.conf 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: Updating smithi149:/etc/ceph/ceph.client.admin.keyring 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:26.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:26 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-04-23T03:32:27.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[61772]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:32:27.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[61772]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:32:27.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[61772]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-23T03:32:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[61772]: Upgrade: It appears safe to stop mon.c 2024-04-23T03:32:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[26683]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:32:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[26683]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:32:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[26683]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-23T03:32:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:27 smithi007 ceph-mon[26683]: Upgrade: It appears safe to stop mon.c 2024-04-23T03:32:27.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:27 smithi149 ceph-mon[22716]: Updating smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:32:27.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:27 smithi149 ceph-mon[22716]: Updating smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-23T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:27 smithi149 ceph-mon[22716]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-04-23T03:32:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:27 smithi149 ceph-mon[22716]: Upgrade: It appears safe to stop mon.c 2024-04-23T03:32:28.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[26683]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:28.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[26683]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:28.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:28.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:28.410 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[26683]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:28.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:28.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:28 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:28.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:28 smithi149 ceph-mon[22716]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:28.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:28 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:28.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:28 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:28.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:28 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:28.662 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:28 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:28.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:29.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-mon[26683]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:32:29.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-mon[26683]: Upgrade: Updating mon.c 2024-04-23T03:32:29.314 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-mon[26683]: Deploying daemon mon.c on smithi007 2024-04-23T03:32:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-mon[61772]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:32:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-mon[61772]: Upgrade: Updating mon.c 2024-04-23T03:32:29.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-mon[61772]: Deploying daemon mon.c on smithi007 2024-04-23T03:32:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 systemd[1]: Stopping Ceph mon.c for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:32:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:29 smithi149 ceph-mon[22716]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-04-23T03:32:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:29 smithi149 ceph-mon[22716]: Upgrade: Updating mon.c 2024-04-23T03:32:29.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:29 smithi149 ceph-mon[22716]: Deploying daemon mon.c on smithi007 2024-04-23T03:32:29.936 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c[26679]: 2024-04-23T03:32:29.584+0000 7f7a0bc74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:32:29.936 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c[26679]: 2024-04-23T03:32:29.584+0000 7f7a0bc74700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-23T03:32:29.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:29 smithi007 podman[64010]: 2024-04-23 03:32:29.666547136 +0000 UTC m=+0.206624162 container died 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, io.k8s.display-name=CentOS Stream 8, name=centos-stream, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, ceph=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-23T03:32:30.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 podman[64010]: 2024-04-23 03:32:30.026373999 +0000 UTC m=+0.566451017 container cleanup 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, architecture=x86_64, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, io.openshift.tags=base centos centos-stream) 2024-04-23T03:32:30.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 bash[64010]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c 2024-04-23T03:32:30.392 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:32:30.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 podman[64024]: 2024-04-23 03:32:30.336121523 +0000 UTC m=+0.669390622 container remove 124daa400b07274bdaa51e90f5f47ef7f9392cb1425d8ba5bdcdec1351cd6d6c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, release=754, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, vendor=Red Hat, Inc., ceph=True, maintainer=Guillaume Abrioux , distribution-scope=public, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-23T03:32:30.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.c.service: Deactivated successfully. 2024-04-23T03:32:30.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 systemd[1]: Stopped Ceph mon.c for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:32:30.704 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.c.service: Consumed 8.225s CPU time. 2024-04-23T03:32:31.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 systemd[1]: Starting Ceph mon.c for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:32:31.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 podman[64206]: 2024-04-23 03:32:30.827955913 +0000 UTC m=+0.018785230 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:32:31.037 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:30 smithi007 podman[64206]: 2024-04-23 03:32:30.937209076 +0000 UTC m=+0.128038364 container create 748d9311f48fa93189f5afeaa1ddfe10c921ac2a86593df4f05ef6a46dcd973f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T03:32:31.120 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (95s) 10s ago 6m 15.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (80s) 10s ago 6m 47.4M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (50s) 10s ago 5m 48.6M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (41s) 10s ago 10m 448M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (75s) 10s ago 12m 473M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (26s) 10s ago 12m 29.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (11m) 10s ago 11m 52.0M 2048M 17.2.0 e1d6a67b021e 63907fd01b38 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (11m) 10s ago 11m 47.1M 2048M 17.2.0 e1d6a67b021e 124daa400b07 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (100s) 10s ago 7m 13.2M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (84s) 10s ago 7m 16.8M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (10m) 10s ago 10m 55.6M 2737M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (10m) 10s ago 10m 56.2M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (9m) 10s ago 9m 49.0M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (9m) 10s ago 9m 53.1M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:32:31.121 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (9m) 10s ago 9m 56.1M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (8m) 10s ago 8m 51.4M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (8m) 10s ago 8m 52.3M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (7m) 10s ago 7m 55.4M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (46s) 10s ago 7m 59.6M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (5m) 10s ago 5m 73.4M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (5m) 10s ago 5m 75.5M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (5m) 10s ago 5m 75.1M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:32:31.122 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (5m) 10s ago 5m 74.7M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 podman[64206]: 2024-04-23 03:32:31.165267086 +0000 UTC m=+0.356096376 container init 748d9311f48fa93189f5afeaa1ddfe10c921ac2a86593df4f05ef6a46dcd973f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=main-56e81df) 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 podman[64206]: 2024-04-23 03:32:31.168748188 +0000 UTC m=+0.359577486 container start 748d9311f48fa93189f5afeaa1ddfe10c921ac2a86593df4f05ef6a46dcd973f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev), process ceph-mon, pid 2 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: pidfile_write: ignore empty --pid-file 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: load: jerasure load: lrc 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: RocksDB version: 7.9.2 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Git sha 0 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Compile date 2024-04-22 15:21:46 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: DB SUMMARY 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: DB Session ID: WBMN1WAFFTGF7A6561Y8 2024-04-23T03:32:31.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: CURRENT file: CURRENT 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 283477 ; 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.error_if_exists: 0 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.create_if_missing: 0 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.paranoid_checks: 1 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.env: 0x558f01347cc0 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.info_log: 0x558f037b7be0 2024-04-23T03:32:31.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T03:32:31.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.statistics: (nil) 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.use_fsync: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_log_file_size: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.allow_fallocate: 1 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.use_direct_reads: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T03:32:31.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.db_log_dir: 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.wal_dir: 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.write_buffer_manager: 0x558f0374b0e0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T03:32:31.337 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.unordered_write: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.row_cache: None 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.wal_filter: None 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.two_write_queues: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T03:32:31.338 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.wal_compression: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.atomic_flush: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.log_readahead_size: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_background_jobs: 2 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_background_compactions: -1 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_subcompactions: 1 2024-04-23T03:32:31.339 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_open_files: -1 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_background_flushes: -1 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Compression algorithms supported: 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kZSTD supported: 0 2024-04-23T03:32:31.340 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kXpressCompression supported: 0 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kZlibCompression supported: 1 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kSnappyCompression supported: 1 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kLZ4Compression supported: 1 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: kBZip2Compression supported: 0 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.merge_operator: 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_filter: None 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T03:32:31.341 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558f037b7dc0) 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_type: 0 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_index_type: 0 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_shortening: 1 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: checksum: 4 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: no_block_cache: 0 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache: 0x558f0377d1f0 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_options: 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: capacity : 536870912 2024-04-23T03:32:31.342 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_shard_bits : 4 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: strict_capacity_limit : 0 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_compressed: (nil) 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: persistent_cache: (nil) 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size: 4096 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size_deviation: 10 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_restart_interval: 16 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_block_restart_interval: 1 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: metadata_block_size: 4096 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: partition_filters: 0 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_delta_encoding: 1 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: filter_policy: bloomfilter 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: whole_key_filtering: 1 2024-04-23T03:32:31.343 INFO:journalctl@ceph.mon.c.smithi007.stdout: verify_compression: 0 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: format_version: 5 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_index_compression: 1 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_align: 0 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: max_auto_readahead_size: 262144 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: prepopulate_block_cache: 0 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: initial_auto_readahead_size: 8192 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression: NoCompression 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T03:32:31.344 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.num_levels: 7 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T03:32:31.345 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T03:32:31.346 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T03:32:31.347 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.inplace_update_support: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.bloom_locality: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.max_successive_merges: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T03:32:31.348 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.ttl: 2592000 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.enable_blob_files: false 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.min_blob_size: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T03:32:31.349 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14986, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6e9b62cf-ad9a-450f-b548-18d9609356d0 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843151199260, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843151200748, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 188993, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14991, "largest_seqno": 15111, "table_properties": {"data_size": 187354, "index_size": 409, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1880, "raw_average_key_size": 26, "raw_value_size": 185714, "raw_average_value_size": 2653, "num_data_blocks": 16, "num_entries": 70, "num_filter_entries": 70, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713843151, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6e9b62cf-ad9a-450f-b548-18d9609356d0", "db_session_id": "WBMN1WAFFTGF7A6561Y8", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843151200835, "job": 1, "event": "recovery_finished"} 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x558f03892000 2024-04-23T03:32:31.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: DB pointer 0x558f0387c000 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** DB Stats ** 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 184.56 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 205.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T03:32:31.351 INFO:journalctl@ceph.mon.c.smithi007.stdout: L6 1/0 11.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 2/0 11.94 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 205.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 205.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 205.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:32:31.352 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 48.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 48.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Block cache BinnedLRUCache@0x558f0377d1f0#2 capacity: 512.00 MB usage: 0.72 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.48 KB,9.23872e-05%) Misc(1,0.00 KB,0%) 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: starting mon.c rank 1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???) e3 preinit fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).mds e1 new map 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).mds e1 print_map 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: e1 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: legacy client fscid: -1 2024-04-23T03:32:31.353 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout: No filesystems configured 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 ceph-mon[64248]: mon.c@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 bash[64206]: 748d9311f48fa93189f5afeaa1ddfe10c921ac2a86593df4f05ef6a46dcd973f 2024-04-23T03:32:31.354 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:31 smithi007 systemd[1]: Started Ceph mon.c for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:32:31.545 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:32:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:32:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-23T03:32:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 1 2024-04-23T03:32:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:32:31.545 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:32:31.546 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:32:31.920 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:32:31.920 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:32:31.920 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:32:31.920 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:32:31.920 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:32:31.921 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:32:31.921 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:32:31.921 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "3/23 daemons upgraded", 2024-04-23T03:32:31.921 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mon daemons", 2024-04-23T03:32:31.921 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:32:31.921 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:32:32.366 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: mon.c calling monitor election 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:32.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: mon.a calling monitor election 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: monmap epoch 3 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: last_changed 2024-04-23T03:21:21.198553+0000 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: min_mon_release 17 (quincy) 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: election_strategy: 1 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: fsmap 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: mgrmap e40: y(active, since 16s), standbys: x 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: overall HEALTH_OK 2024-04-23T03:32:32.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:32.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:32.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:32.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:32.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:32.924 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:32 smithi149 ceph-mon[22716]: from='client.? 172.21.15.7:0/4032923179' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: mon.c calling monitor election 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: mon.a calling monitor election 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:33.014 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: monmap epoch 3 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: last_changed 2024-04-23T03:21:21.198553+0000 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: min_mon_release 17 (quincy) 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: election_strategy: 1 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: fsmap 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: mgrmap e40: y(active, since 16s), standbys: x 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: overall HEALTH_OK 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:33.015 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/4032923179' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='client.44136 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='client.44142 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='client.44148 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: mon.c calling monitor election 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: mon.a calling monitor election 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: monmap epoch 3 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: last_changed 2024-04-23T03:21:21.198553+0000 2024-04-23T03:32:33.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: min_mon_release 17 (quincy) 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: election_strategy: 1 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: fsmap 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: mgrmap e40: y(active, since 16s), standbys: x 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: overall HEALTH_OK 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='client.44160 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:32:33.017 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:32 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/4032923179' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:32:33.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:34.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:34 smithi149 ceph-mon[22716]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:34.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:34 smithi007 ceph-mon[64248]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:34.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:34 smithi007 ceph-mon[61772]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-04-23T03:32:36.467 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:32:36] "GET /metrics HTTP/1.1" 200 38178 "" "Prometheus/2.33.4" 2024-04-23T03:32:36.467 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:36.209+0000 7f1c772de700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-23T03:32:37.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:37 smithi007 ceph-mon[61772]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:37.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:37 smithi007 ceph-mon[64248]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:37.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:37 smithi149 ceph-mon[22716]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:37 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:37 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:38 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:38 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:38 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:38 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:38.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:38 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:38 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:38 smithi149 ceph-mon[22716]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:38 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:38 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:39 smithi007 ceph-mon[61772]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:39.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:39 smithi007 ceph-mon[64248]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:39.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:39 smithi149 ceph-mon[22716]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:41.448 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:41.449 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:41 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-23T03:32:41.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:41.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:41.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:41.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:32:41.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:32:41.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-04-23T03:32:41.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:41 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-04-23T03:32:42.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:42 smithi149 ceph-mon[22716]: Upgrade: It appears safe to stop mon.b 2024-04-23T03:32:42.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:42 smithi007 ceph-mon[61772]: Upgrade: It appears safe to stop mon.b 2024-04-23T03:32:42.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:42 smithi007 ceph-mon[64248]: Upgrade: It appears safe to stop mon.b 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: Upgrade: Updating mon.b 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-mon[22716]: Deploying daemon mon.b on smithi149 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: Upgrade: Updating mon.b 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[61772]: Deploying daemon mon.b on smithi149 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: Upgrade: Updating mon.b 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:43.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:43.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:43.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:43 smithi007 ceph-mon[64248]: Deploying daemon mon.b on smithi149 2024-04-23T03:32:43.789 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:44.341 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 systemd[1]: Stopping Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:32:44.341 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b[22712]: 2024-04-23T03:32:44.318+0000 7f1e99ad8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:32:44.341 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b[22712]: 2024-04-23T03:32:44.318+0000 7f1e99ad8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-23T03:32:44.611 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 podman[54679]: 2024-04-23 03:32:44.339278623 +0000 UTC m=+0.111020775 container died 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, CEPH_POINT_RELEASE=-17.2.0) 2024-04-23T03:32:44.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:44] ENGINE Bus STOPPING 2024-04-23T03:32:44.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:32:44.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:44] ENGINE Bus STOPPED 2024-04-23T03:32:44.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:44] ENGINE Bus STARTING 2024-04-23T03:32:44.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:44] ENGINE Serving on http://:::9283 2024-04-23T03:32:44.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:32:44] ENGINE Bus STARTED 2024-04-23T03:32:44.878 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 podman[54679]: 2024-04-23 03:32:44.610488129 +0000 UTC m=+0.382230280 container cleanup 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, RELEASE=HEAD, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, release=754, version=8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-23T03:32:44.879 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 bash[54679]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b 2024-04-23T03:32:45.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:44 smithi149 podman[54694]: 2024-04-23 03:32:44.878478982 +0000 UTC m=+0.533399077 container remove 63907fd01b380c51a1bf9859b316937f707f5c14afd1c35f3e560bcfd249826d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, ceph=True, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, maintainer=Guillaume Abrioux , release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-04-23T03:32:45.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-23T03:32:45.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 systemd[1]: Stopped Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:32:45.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.b.service: Consumed 9.618s CPU time. 2024-04-23T03:32:45.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 systemd[1]: Starting Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:32:45.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 podman[54829]: 2024-04-23 03:32:45.356762021 +0000 UTC m=+0.018495236 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:32:45.602 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 podman[54829]: 2024-04-23 03:32:45.472049711 +0000 UTC m=+0.133782915 container create 896e9f535de0c41ac76717ea6f8829cb81feba99e0ed3613e8d44a651b5e357e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-56e81df) 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 podman[54829]: 2024-04-23 03:32:45.724791814 +0000 UTC m=+0.386525019 container init 896e9f535de0c41ac76717ea6f8829cb81feba99e0ed3613e8d44a651b5e357e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=main-56e81df) 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 podman[54829]: 2024-04-23 03:32:45.727819393 +0000 UTC m=+0.389552598 container start 896e9f535de0c41ac76717ea6f8829cb81feba99e0ed3613e8d44a651b5e357e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: set uid:gid to 167:167 (ceph:ceph) 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev), process ceph-mon, pid 2 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: pidfile_write: ignore empty --pid-file 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: load: jerasure load: lrc 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: RocksDB version: 7.9.2 2024-04-23T03:32:46.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Git sha 0 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Compile date 2024-04-22 15:21:46 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: DB SUMMARY 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: DB Session ID: DXY7RYZRXKKZ2L4IWO21 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: CURRENT file: CURRENT 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: IDENTITY file: IDENTITY 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1092 Bytes 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1019228 ; 2024-04-23T03:32:46.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.error_if_exists: 0 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.create_if_missing: 0 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.paranoid_checks: 1 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.env: 0x559da1e28cc0 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.fs: PosixFileSystem 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.info_log: 0x559da332fbe0 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_file_opening_threads: 16 2024-04-23T03:32:46.175 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.statistics: (nil) 2024-04-23T03:32:46.177 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.use_fsync: 0 2024-04-23T03:32:46.177 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_log_file_size: 0 2024-04-23T03:32:46.177 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-23T03:32:46.177 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.keep_log_file_num: 1000 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.recycle_log_file_num: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.allow_fallocate: 1 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.allow_mmap_reads: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.allow_mmap_writes: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.use_direct_reads: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.create_missing_column_families: 0 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.db_log_dir: 2024-04-23T03:32:46.178 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.wal_dir: 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.advise_random_on_open: 1 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.db_write_buffer_size: 0 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.write_buffer_manager: 0x559da32c30e0 2024-04-23T03:32:46.179 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.rate_limiter: (nil) 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.wal_recovery_mode: 2 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.enable_thread_tracking: 0 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.enable_pipelined_write: 0 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.unordered_write: 0 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-23T03:32:46.180 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.row_cache: None 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.wal_filter: None 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.allow_ingest_behind: 0 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.two_write_queues: 0 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.manual_wal_flush: 0 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.wal_compression: 0 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.atomic_flush: 0 2024-04-23T03:32:46.181 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.log_readahead_size: 0 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.best_efforts_recovery: 0 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.allow_data_in_errors: 0 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.db_host_id: __hostname__ 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-23T03:32:46.182 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_background_jobs: 2 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_background_compactions: -1 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_subcompactions: 1 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_total_wal_size: 0 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-23T03:32:46.183 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_open_files: -1 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bytes_per_sync: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_readahead_size: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_background_flushes: -1 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Compression algorithms supported: 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kZSTD supported: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kXpressCompression supported: 0 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kLZ4HCCompression supported: 1 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kZlibCompression supported: 1 2024-04-23T03:32:46.184 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kSnappyCompression supported: 1 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kLZ4Compression supported: 1 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: kBZip2Compression supported: 0 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.merge_operator: 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_filter: None 2024-04-23T03:32:46.185 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_filter_factory: None 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.sst_partitioner_factory: None 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559da332fdc0) 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks: 1 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: pin_top_level_index_and_filter: 1 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_type: 0 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_index_type: 0 2024-04-23T03:32:46.186 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_shortening: 1 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: checksum: 4 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: no_block_cache: 0 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache: 0x559da32f51f0 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_name: BinnedLRUCache 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_options: 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: capacity : 536870912 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_shard_bits : 4 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: strict_capacity_limit : 0 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: high_pri_pool_ratio: 0.000 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_cache_compressed: (nil) 2024-04-23T03:32:46.187 INFO:journalctl@ceph.mon.b.smithi149.stdout: persistent_cache: (nil) 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size: 4096 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_size_deviation: 10 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_restart_interval: 16 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: index_block_restart_interval: 1 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: metadata_block_size: 4096 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: partition_filters: 0 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: use_delta_encoding: 1 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: filter_policy: bloomfilter 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: whole_key_filtering: 1 2024-04-23T03:32:46.188 INFO:journalctl@ceph.mon.b.smithi149.stdout: verify_compression: 0 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: read_amp_bytes_per_bit: 0 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: format_version: 5 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_index_compression: 1 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: block_align: 0 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: max_auto_readahead_size: 262144 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: prepopulate_block_cache: 0 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: initial_auto_readahead_size: 8192 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout: num_file_reads_for_auto_readahead: 2 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.write_buffer_size: 33554432 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_write_buffer_number: 2 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression: NoCompression 2024-04-23T03:32:46.189 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression: Disabled 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.prefix_extractor: nullptr 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.num_levels: 7 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-23T03:32:46.190 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.level: 32767 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.strategy: 0 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-23T03:32:46.191 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.enabled: false 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.target_file_size_base: 67108864 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-23T03:32:46.192 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.arena_block_size: 1048576 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-23T03:32:46.193 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.disable_auto_compactions: 0 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-23T03:32:46.194 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.inplace_update_support: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.bloom_locality: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.max_successive_merges: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.paranoid_file_checks: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.force_consistency_checks: 1 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.report_bg_io_stats: 0 2024-04-23T03:32:46.195 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.ttl: 2592000 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.enable_blob_files: false 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.min_blob_size: 0 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.blob_file_size: 268435456 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-23T03:32:46.196 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.blob_file_starting_level: 0 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14891, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: a7867ccb-6e6e-4bf4-bb06-8a91defbe9ce 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843165757537, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843165760701, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 609787, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14896, "largest_seqno": 15307, "table_properties": {"data_size": 607023, "index_size": 1212, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4704, "raw_average_key_size": 24, "raw_value_size": 602783, "raw_average_value_size": 3172, "num_data_blocks": 54, "num_entries": 190, "num_filter_entries": 190, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713843165, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a7867ccb-6e6e-4bf4-bb06-8a91defbe9ce", "db_session_id": "DXY7RYZRXKKZ2L4IWO21", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-04-23T03:32:46.197 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713843165760788, "job": 1, "event": "recovery_finished"} 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559da340a000 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: DB pointer 0x559da33f4000 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** DB Stats ** 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:46.198 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: L0 1/0 595.50 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 340.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: L6 1/0 11.76 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Sum 2/0 12.34 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 340.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-23T03:32:46.199 INFO:journalctl@ceph.mon.b.smithi149.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 340.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** Compaction Stats [default] ** 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 340.1 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-23T03:32:46.200 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: Cumulative compaction: 0.00 GB write, 110.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: Interval compaction: 0.00 GB write, 110.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache BinnedLRUCache@0x559da32f51f0#2 capacity: 512.00 MB usage: 1.92 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.33 KB,0.00025332%) Misc(1,0.00 KB,0%) 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: starting mon.b rank 2 at public addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] at bind addrs [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???) e3 preinit fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:46.201 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).mds e1 new map 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).mds e1 print_map 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout: e1 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout: legacy client fscid: -1 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout: 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout: No filesystems configured 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:46.202 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: mon.b@-1(???).paxosservice(auth 1..25) refresh upgraded, format 0 -> 3 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 bash[54829]: 896e9f535de0c41ac76717ea6f8829cb81feba99e0ed3613e8d44a651b5e357e 2024-04-23T03:32:46.203 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:45 smithi149 systemd[1]: Started Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:32:46.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:32:46] "GET /metrics HTTP/1.1" 200 38186 "" "Prometheus/2.33.4" 2024-04-23T03:32:46.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[64248]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:46.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:46.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:46.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:46.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[61772]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:46.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:46.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:46.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:45 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: fsmap 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: mgrmap e40: y(active, since 30s), standbys: x 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: overall HEALTH_OK 2024-04-23T03:32:47.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: mon.a calling monitor election 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: mon.b calling monitor election 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: mon.c calling monitor election 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: monmap epoch 4 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: last_changed 2024-04-23T03:32:45.843816+0000 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: min_mon_release 19 (squid) 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: election_strategy: 1 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: fsmap 2024-04-23T03:32:47.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:47.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: mgrmap e40: y(active, since 30s), standbys: x 2024-04-23T03:32:47.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: overall HEALTH_OK 2024-04-23T03:32:47.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:47.174 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: fsmap 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: mgrmap e40: y(active, since 30s), standbys: x 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: overall HEALTH_OK 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:47.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: mon.a calling monitor election 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: mon.b calling monitor election 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: mon.c calling monitor election 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: monmap epoch 4 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: last_changed 2024-04-23T03:32:45.843816+0000 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: min_mon_release 19 (squid) 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: election_strategy: 1 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: fsmap 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: mgrmap e40: y(active, since 30s), standbys: x 2024-04-23T03:32:47.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: overall HEALTH_OK 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: fsmap 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: mgrmap e40: y(active, since 30s), standbys: x 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: overall HEALTH_OK 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: mon.a calling monitor election 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: mon.b calling monitor election 2024-04-23T03:32:47.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: mon.c calling monitor election 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: monmap epoch 4 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: fsid 3d05926e-0120-11ef-bc93-c7b262605968 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: last_changed 2024-04-23T03:32:45.843816+0000 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: created 2024-04-23T03:19:36.264646+0000 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: min_mon_release 19 (squid) 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: election_strategy: 1 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: 0: [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon.a 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: 1: [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon.c 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: 2: [v2:172.21.15.149:3300/0,v1:172.21.15.149:6789/0] mon.b 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: fsmap 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: osdmap e105: 8 total, 8 up, 8 in 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: mgrmap e40: y(active, since 30s), standbys: x 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: overall HEALTH_OK 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:47.336 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:48.154 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:47 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:48.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:47 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:48.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:47 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:48.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:48 smithi149 ceph-mon[54848]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:48.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:49.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:48 smithi007 ceph-mon[61772]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:49.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:48 smithi007 ceph-mon[64248]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:51.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:32:50.768+0000 7f1c772de700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-23T03:32:51.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-mon[61772]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:51.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:51.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:51.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-mon[64248]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:51.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:51.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:50 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:51.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:50 smithi149 ceph-mon[54848]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:50 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:50 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:52.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:52 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:52.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:52 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:52.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:52 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:52.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:52 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:52 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:52.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:52 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:53.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:53 smithi149 ceph-mon[54848]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:53.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:53 smithi007 ceph-mon[64248]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:53.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:53 smithi007 ceph-mon[61772]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:53.816 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:55.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:55.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:55 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:55.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:55.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:55.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:32:55.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:55.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:55.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:55.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:55 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:56.211 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: Reconfiguring mon.a (monmap changed)... 2024-04-23T03:32:56.211 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: Reconfiguring daemon mon.a on smithi007 2024-04-23T03:32:56.211 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:32:56] "GET /metrics HTTP/1.1" 200 38186 "" "Prometheus/2.33.4" 2024-04-23T03:32:56.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: Reconfiguring mon.a (monmap changed)... 2024-04-23T03:32:56.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: Reconfiguring daemon mon.a on smithi007 2024-04-23T03:32:56.487 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:32:56.488 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:32:56.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:32:56.489 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:56 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: Reconfiguring mon.a (monmap changed)... 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: Reconfiguring daemon mon.a on smithi007 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:32:56.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:56 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:57.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:57 smithi007 ceph-mon[64248]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:57.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:57 smithi007 ceph-mon[64248]: Reconfiguring mgr.y (monmap changed)... 2024-04-23T03:32:57.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:57 smithi007 ceph-mon[64248]: Reconfiguring daemon mgr.y on smithi007 2024-04-23T03:32:57.494 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:57 smithi007 ceph-mon[61772]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:57.494 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:57 smithi007 ceph-mon[61772]: Reconfiguring mgr.y (monmap changed)... 2024-04-23T03:32:57.494 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:57 smithi007 ceph-mon[61772]: Reconfiguring daemon mgr.y on smithi007 2024-04-23T03:32:57.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:57 smithi149 ceph-mon[54848]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:32:57.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:57 smithi149 ceph-mon[54848]: Reconfiguring mgr.y (monmap changed)... 2024-04-23T03:32:57.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:57 smithi149 ceph-mon[54848]: Reconfiguring daemon mgr.y on smithi007 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:58.522 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:58.523 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:58.523 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:58.523 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:58.523 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:58.523 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:58 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:58.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:32:58.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:58.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:32:58.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:32:58.662 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:32:58.662 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:32:58.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:32:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:32:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:32:59.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:59 smithi007 ceph-mon[61772]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:59.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:59 smithi007 ceph-mon[61772]: Reconfiguring mon.c (monmap changed)... 2024-04-23T03:32:59.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:32:59 smithi007 ceph-mon[61772]: Reconfiguring daemon mon.c on smithi007 2024-04-23T03:32:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:59 smithi007 ceph-mon[64248]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:59 smithi007 ceph-mon[64248]: Reconfiguring mon.c (monmap changed)... 2024-04-23T03:32:59.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:32:59 smithi007 ceph-mon[64248]: Reconfiguring daemon mon.c on smithi007 2024-04-23T03:32:59.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:59 smithi149 ceph-mon[54848]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:32:59.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:59 smithi149 ceph-mon[54848]: Reconfiguring mon.c (monmap changed)... 2024-04-23T03:32:59.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:32:59 smithi149 ceph-mon[54848]: Reconfiguring daemon mon.c on smithi007 2024-04-23T03:33:00.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:00.533 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:00.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[64248]: Reconfiguring osd.0 (monmap changed)... 2024-04-23T03:33:00.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:33:00.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:00.534 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.0 on smithi007 2024-04-23T03:33:00.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:00.534 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:00.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[61772]: Reconfiguring osd.0 (monmap changed)... 2024-04-23T03:33:00.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:33:00.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:00.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:00 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.0 on smithi007 2024-04-23T03:33:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:00 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:00 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:00 smithi149 ceph-mon[54848]: Reconfiguring osd.0 (monmap changed)... 2024-04-23T03:33:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:00 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:33:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:00 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:00 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.0 on smithi007 2024-04-23T03:33:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[64248]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[61772]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:33:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:01.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:01 smithi149 ceph-mon[54848]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:01 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:01 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:33:01.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:02.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:02 smithi007 ceph-mon[61772]: Reconfiguring osd.1 (monmap changed)... 2024-04-23T03:33:02.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:02 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.1 on smithi007 2024-04-23T03:33:02.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:02 smithi007 ceph-mon[64248]: Reconfiguring osd.1 (monmap changed)... 2024-04-23T03:33:02.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:02 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.1 on smithi007 2024-04-23T03:33:02.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:02 smithi149 ceph-mon[54848]: Reconfiguring osd.1 (monmap changed)... 2024-04-23T03:33:02.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:02 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.1 on smithi007 2024-04-23T03:33:02.741 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (2m) 27s ago 6m 15.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (113s) 13s ago 6m 47.6M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (82s) 27s ago 5m 49.5M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (73s) 13s ago 11m 450M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (107s) 27s ago 13m 493M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (58s) 27s ago 13m 40.8M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (18s) 13s ago 11m 18.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (32s) 27s ago 11m 18.3M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (2m) 27s ago 7m 17.8M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (117s) 13s ago 7m 17.0M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (11m) 27s ago 11m 55.8M 2737M 17.2.0 e1d6a67b021e 3450caffb700 2024-04-23T03:33:03.457 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (10m) 27s ago 10m 56.4M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (10m) 27s ago 10m 49.5M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (9m) 27s ago 9m 53.2M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (9m) 13s ago 9m 56.4M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (9m) 13s ago 9m 51.8M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (8m) 13s ago 8m 52.7M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (8m) 13s ago 8m 55.9M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (78s) 13s ago 7m 60.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (6m) 27s ago 6m 73.6M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (6m) 13s ago 6m 75.9M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (6m) 27s ago 6m 75.2M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:33:03.458 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (6m) 13s ago 6m 75.1M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:33:03.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:03 smithi149 ceph-mon[54848]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:03.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:03 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:03.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:03 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:03.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:03 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:33:03.507 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:03 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[64248]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[61772]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:03.517 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:33:03.518 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:03 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:03.877 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:03.878 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:33:03.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-04-23T03:33:03.879 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 5 2024-04-23T03:33:03.879 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:33:03.879 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:33:03.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:03.506Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:04.231 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:33:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:33:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:33:04.231 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "5/23 daemons upgraded", 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading mon daemons", 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:33:04.232 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:33:04.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[64248]: from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:04.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[64248]: Reconfiguring osd.2 (monmap changed)... 2024-04-23T03:33:04.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.2 on smithi007 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[64248]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/139663241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[61772]: from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[61772]: Reconfiguring osd.2 (monmap changed)... 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.2 on smithi007 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[61772]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:04.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:04 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/139663241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:04.659 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:33:04.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:04 smithi149 ceph-mon[54848]: from='client.54103 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:04 smithi149 ceph-mon[54848]: Reconfiguring osd.2 (monmap changed)... 2024-04-23T03:33:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:04 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.2 on smithi007 2024-04-23T03:33:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:04 smithi149 ceph-mon[54848]: from='client.44178 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:04 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/139663241' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='client.44184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/613326261' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='client.44184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:05.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:05 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/613326261' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:33:05.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='client.44184 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='client.54127 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:05.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:05 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/613326261' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:33:06.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:33:06] "GET /metrics HTTP/1.1" 200 38257 "" "Prometheus/2.33.4" 2024-04-23T03:33:06.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: Reconfiguring osd.3 (monmap changed)... 2024-04-23T03:33:06.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.3 on smithi007 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: Reconfiguring osd.3 (monmap changed)... 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.3 on smithi007 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:06.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:06 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: Reconfiguring osd.3 (monmap changed)... 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.3 on smithi007 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:06.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:06 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:07.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:07 smithi007 ceph-mon[61772]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:07.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:07 smithi007 ceph-mon[61772]: Reconfiguring rgw.foo.smithi007.bipufo (monmap changed)... 2024-04-23T03:33:07.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:07 smithi007 ceph-mon[61772]: Reconfiguring daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:33:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:07 smithi007 ceph-mon[64248]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:07 smithi007 ceph-mon[64248]: Reconfiguring rgw.foo.smithi007.bipufo (monmap changed)... 2024-04-23T03:33:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:07 smithi007 ceph-mon[64248]: Reconfiguring daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:33:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:07 smithi149 ceph-mon[54848]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:07 smithi149 ceph-mon[54848]: Reconfiguring rgw.foo.smithi007.bipufo (monmap changed)... 2024-04-23T03:33:07.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:07 smithi149 ceph-mon[54848]: Reconfiguring daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:33:08.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:08.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:08.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:08.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:08 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:08.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:08.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:08.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:08.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:08.662 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:08.662 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:08.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:09 smithi007 ceph-mon[64248]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:09 smithi007 ceph-mon[64248]: Reconfiguring rgw.smpl.smithi007.mxntqi (monmap changed)... 2024-04-23T03:33:09.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:09 smithi007 ceph-mon[64248]: Reconfiguring daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:33:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:09 smithi007 ceph-mon[61772]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:09 smithi007 ceph-mon[61772]: Reconfiguring rgw.smpl.smithi007.mxntqi (monmap changed)... 2024-04-23T03:33:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:09 smithi007 ceph-mon[61772]: Reconfiguring daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:33:09.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:09 smithi149 ceph-mon[54848]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:09.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:09 smithi149 ceph-mon[54848]: Reconfiguring rgw.smpl.smithi007.mxntqi (monmap changed)... 2024-04-23T03:33:09.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:09 smithi149 ceph-mon[54848]: Reconfiguring daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:33:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:10.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: Reconfiguring mon.b (monmap changed)... 2024-04-23T03:33:10.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:33:10.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:33:10.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:10.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:10 smithi149 ceph-mon[54848]: Reconfiguring daemon mon.b on smithi149 2024-04-23T03:33:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: Reconfiguring mon.b (monmap changed)... 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[64248]: Reconfiguring daemon mon.b on smithi149 2024-04-23T03:33:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: Reconfiguring mon.b (monmap changed)... 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:10.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:10 smithi007 ceph-mon[61772]: Reconfiguring daemon mon.b on smithi149 2024-04-23T03:33:12.255 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: Reconfiguring mgr.x (monmap changed)... 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:12.256 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:11 smithi149 ceph-mon[54848]: Reconfiguring daemon mgr.x on smithi149 2024-04-23T03:33:12.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:12.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:12.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: Reconfiguring mgr.x (monmap changed)... 2024-04-23T03:33:12.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[64248]: Reconfiguring daemon mgr.x on smithi149 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: Reconfiguring mgr.x (monmap changed)... 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-23T03:33:12.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:12.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:11 smithi007 ceph-mon[61772]: Reconfiguring daemon mgr.x on smithi149 2024-04-23T03:33:13.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:13 smithi149 ceph-mon[54848]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:13.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:33:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:13.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[64248]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:13.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:13.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[61772]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:33:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:13.782 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:14.095 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:14 smithi149 ceph-mon[54848]: Reconfiguring osd.4 (monmap changed)... 2024-04-23T03:33:14.095 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:14 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.4 on smithi149 2024-04-23T03:33:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:14 smithi007 ceph-mon[64248]: Reconfiguring osd.4 (monmap changed)... 2024-04-23T03:33:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:14 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.4 on smithi149 2024-04-23T03:33:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:14 smithi007 ceph-mon[61772]: Reconfiguring osd.4 (monmap changed)... 2024-04-23T03:33:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:14 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.4 on smithi149 2024-04-23T03:33:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: Reconfiguring osd.5 (monmap changed)... 2024-04-23T03:33:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:33:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:15.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:15 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.5 on smithi149 2024-04-23T03:33:15.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: Reconfiguring osd.5 (monmap changed)... 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.5 on smithi149 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: Reconfiguring osd.5 (monmap changed)... 2024-04-23T03:33:15.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:33:15.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:15.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:15 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.5 on smithi149 2024-04-23T03:33:16.455 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:16.456 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:16.456 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:16.456 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:33:16.456 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:16.507 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:33:16] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.33.4" 2024-04-23T03:33:16.508 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:16.508 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:16.508 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:33:16.509 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: Reconfiguring osd.6 (monmap changed)... 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.6 on smithi149 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: Reconfiguring osd.6 (monmap changed)... 2024-04-23T03:33:17.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.6 on smithi149 2024-04-23T03:33:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:33:17.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:17 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:17.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:17.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: Reconfiguring osd.6 (monmap changed)... 2024-04-23T03:33:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.6 on smithi149 2024-04-23T03:33:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:33:17.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:17 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:18 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:18 smithi149 ceph-mon[54848]: Reconfiguring osd.7 (monmap changed)... 2024-04-23T03:33:18.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:18 smithi149 ceph-mon[54848]: Reconfiguring daemon osd.7 on smithi149 2024-04-23T03:33:18.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:18 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:18.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:18 smithi007 ceph-mon[64248]: Reconfiguring osd.7 (monmap changed)... 2024-04-23T03:33:18.478 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:18 smithi007 ceph-mon[64248]: Reconfiguring daemon osd.7 on smithi149 2024-04-23T03:33:18.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:18 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:18.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:18 smithi007 ceph-mon[61772]: Reconfiguring osd.7 (monmap changed)... 2024-04-23T03:33:18.479 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:18 smithi007 ceph-mon[61772]: Reconfiguring daemon osd.7 on smithi149 2024-04-23T03:33:18.763 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:19.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[64248]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[61772]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:19.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:19 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:19.452 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:19 smithi149 ceph-mon[54848]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:19.452 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:19 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:19.452 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:19 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:19.452 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:19 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:19.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:19 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:19.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:19 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: Reconfiguring rgw.foo.smithi149.axrzpl (monmap changed)... 2024-04-23T03:33:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: Reconfiguring daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:33:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:20.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:20.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:20 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:20.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: Reconfiguring rgw.foo.smithi149.axrzpl (monmap changed)... 2024-04-23T03:33:20.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: Reconfiguring daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: Reconfiguring rgw.foo.smithi149.axrzpl (monmap changed)... 2024-04-23T03:33:20.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: Reconfiguring daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:33:20.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:20.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:20.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:20.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:33:20.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:20 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:21.453 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:21 smithi149 ceph-mon[54848]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:21.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:21 smithi149 ceph-mon[54848]: Reconfiguring rgw.smpl.smithi149.rtdeuy (monmap changed)... 2024-04-23T03:33:21.454 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:21 smithi149 ceph-mon[54848]: Reconfiguring daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:33:21.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:21 smithi007 ceph-mon[64248]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:21.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:21 smithi007 ceph-mon[64248]: Reconfiguring rgw.smpl.smithi149.rtdeuy (monmap changed)... 2024-04-23T03:33:21.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:21 smithi007 ceph-mon[64248]: Reconfiguring daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:33:21.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:21 smithi007 ceph-mon[61772]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:21 smithi007 ceph-mon[61772]: Reconfiguring rgw.smpl.smithi149.rtdeuy (monmap changed)... 2024-04-23T03:33:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:21 smithi007 ceph-mon[61772]: Reconfiguring daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:33:22.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all mon 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-23T03:33:22.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all crash 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[64248]: Upgrade: osd.0 is safe to restart 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all mon 2024-04-23T03:33:22.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all crash 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-23T03:33:22.835 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:22 smithi007 ceph-mon[61772]: Upgrade: osd.0 is safe to restart 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all mon 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-04-23T03:33:22.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all crash 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-04-23T03:33:22.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:22 smithi149 ceph-mon[54848]: Upgrade: osd.0 is safe to restart 2024-04-23T03:33:23.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:23 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:24.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[64248]: Upgrade: Updating osd.0 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[64248]: Deploying daemon osd.0 on smithi007 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[64248]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[61772]: Upgrade: Updating osd.0 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[61772]: Deploying daemon osd.0 on smithi007 2024-04-23T03:33:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:24 smithi007 ceph-mon[61772]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:24 smithi149 ceph-mon[54848]: Upgrade: Updating osd.0 2024-04-23T03:33:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:24 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:24 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-23T03:33:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:24 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:24 smithi149 ceph-mon[54848]: Deploying daemon osd.0 on smithi007 2024-04-23T03:33:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:24 smithi149 ceph-mon[54848]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:26.423 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:33:26] "GET /metrics HTTP/1.1" 200 38253 "" "Prometheus/2.33.4" 2024-04-23T03:33:26.825 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:26 smithi007 ceph-mon[64248]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:26.825 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:26 smithi007 ceph-mon[61772]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:26.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:26 smithi149 ceph-mon[54848]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:27.082 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:26 smithi007 systemd[1]: Stopping Ceph osd.0 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:33:27.532 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[29530]: 2024-04-23T03:33:27.088+0000 7fecaa279700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:33:27.532 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[29530]: 2024-04-23T03:33:27.088+0000 7fecaa279700 -1 osd.0 105 *** Got signal Terminated *** 2024-04-23T03:33:27.532 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[29530]: 2024-04-23T03:33:27.088+0000 7fecaa279700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:33:27.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:27.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-mon[64248]: osd.0 marked itself down and dead 2024-04-23T03:33:27.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:27 smithi007 ceph-mon[61772]: osd.0 marked itself down and dead 2024-04-23T03:33:27.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:27 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:27.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:27 smithi149 ceph-mon[54848]: osd.0 marked itself down and dead 2024-04-23T03:33:28.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:27 smithi007 podman[67076]: 2024-04-23 03:33:27.873323258 +0000 UTC m=+0.896062834 container stop 3450caffb70058dddd10fed954f90fabf5c12a65dc9878d6317c3fe740c7a1ee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, vcs-type=git, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_BRANCH=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8) 2024-04-23T03:33:28.179 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:27 smithi007 podman[67076]: 2024-04-23 03:33:27.89914592 +0000 UTC m=+0.921885490 container died 3450caffb70058dddd10fed954f90fabf5c12a65dc9878d6317c3fe740c7a1ee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_CLEAN=True, name=centos-stream, version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-23T03:33:28.454 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:28 smithi007 podman[67076]: 2024-04-23 03:33:28.286364868 +0000 UTC m=+1.309104437 container cleanup 3450caffb70058dddd10fed954f90fabf5c12a65dc9878d6317c3fe740c7a1ee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, GIT_CLEAN=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, architecture=x86_64, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, vcs-type=git, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc.) 2024-04-23T03:33:28.454 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:28 smithi007 bash[67076]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0 2024-04-23T03:33:28.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:28 smithi007 ceph-mon[64248]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:28.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:28 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:33:28.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:28 smithi007 ceph-mon[64248]: osdmap e106: 8 total, 7 up, 8 in 2024-04-23T03:33:28.833 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:28 smithi007 podman[67104]: 2024-04-23 03:33:28.454153518 +0000 UTC m=+0.574666621 container remove 3450caffb70058dddd10fed954f90fabf5c12a65dc9878d6317c3fe740c7a1ee (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_BRANCH=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-23T03:33:28.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:28 smithi007 ceph-mon[61772]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:28.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:28 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:33:28.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:28 smithi007 ceph-mon[61772]: osdmap e106: 8 total, 7 up, 8 in 2024-04-23T03:33:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:28 smithi149 ceph-mon[54848]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:28 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:33:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:28 smithi149 ceph-mon[54848]: osdmap e106: 8 total, 7 up, 8 in 2024-04-23T03:33:28.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:29.191 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:28 smithi007 podman[67191]: 2024-04-23 03:33:28.818886876 +0000 UTC m=+0.018898134 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:33:29.191 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:28 smithi007 podman[67191]: 2024-04-23 03:33:28.955369765 +0000 UTC m=+0.155381023 container create 9b0ab74bb8843fe23331aa12ed66d6f499ac0de325673a29ea913c8147ecb4f0 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416) 2024-04-23T03:33:29.500 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:29 smithi007 podman[67191]: 2024-04-23 03:33:29.191408285 +0000 UTC m=+0.391419543 container init 9b0ab74bb8843fe23331aa12ed66d6f499ac0de325673a29ea913c8147ecb4f0 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2) 2024-04-23T03:33:29.500 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:29 smithi007 podman[67191]: 2024-04-23 03:33:29.194708129 +0000 UTC m=+0.394719396 container start 9b0ab74bb8843fe23331aa12ed66d6f499ac0de325673a29ea913c8147ecb4f0 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df) 2024-04-23T03:33:29.500 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:29 smithi007 podman[67191]: 2024-04-23 03:33:29.28311648 +0000 UTC m=+0.483127762 container attach 9b0ab74bb8843fe23331aa12ed66d6f499ac0de325673a29ea913c8147ecb4f0 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-56e81df, GIT_CLEAN=True) 2024-04-23T03:33:29.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:29 smithi007 ceph-mon[64248]: osdmap e107: 8 total, 7 up, 8 in 2024-04-23T03:33:29.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:29 smithi007 ceph-mon[61772]: osdmap e107: 8 total, 7 up, 8 in 2024-04-23T03:33:29.833 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:29 smithi007 podman[67191]: 2024-04-23 03:33:29.500282132 +0000 UTC m=+0.700293394 container died 9b0ab74bb8843fe23331aa12ed66d6f499ac0de325673a29ea913c8147ecb4f0 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:33:29.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:29 smithi149 ceph-mon[54848]: osdmap e107: 8 total, 7 up, 8 in 2024-04-23T03:33:30.596 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:30 smithi007 ceph-mon[64248]: pgmap v42: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:33:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:30 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:30.597 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:30 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:30.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:30 smithi007 ceph-mon[61772]: pgmap v42: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:33:30.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:30 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:30.598 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:30 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:30.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 podman[67217]: 2024-04-23 03:33:30.345882039 +0000 UTC m=+0.839647260 container remove 9b0ab74bb8843fe23331aa12ed66d6f499ac0de325673a29ea913c8147ecb4f0 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-23T03:33:30.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-23T03:33:30.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 systemd[1]: Stopped Ceph osd.0 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:33:30.598 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.0.service: Consumed 7.692s CPU time. 2024-04-23T03:33:30.599 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 systemd[1]: Starting Ceph osd.0 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:33:30.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:30 smithi149 ceph-mon[54848]: pgmap v42: 161 pgs: 21 stale+active+clean, 140 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:33:30.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:30 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:30.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:30 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:31.075 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 podman[67327]: 2024-04-23 03:33:30.717070805 +0000 UTC m=+0.018916799 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:33:31.075 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:30 smithi007 podman[67327]: 2024-04-23 03:33:30.838509299 +0000 UTC m=+0.140355294 container create 2f1420af7749dacf84f22abbcd24e391c6b1ccb0e521effb7d3769f16c3f9f2b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T03:33:31.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 podman[67327]: 2024-04-23 03:33:31.074545874 +0000 UTC m=+0.376391867 container init 2f1420af7749dacf84f22abbcd24e391c6b1ccb0e521effb7d3769f16c3f9f2b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-23T03:33:31.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 podman[67327]: 2024-04-23 03:33:31.078271261 +0000 UTC m=+0.380117261 container start 2f1420af7749dacf84f22abbcd24e391c6b1ccb0e521effb7d3769f16c3f9f2b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, ceph=True) 2024-04-23T03:33:31.333 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 podman[67327]: 2024-04-23 03:33:31.166294317 +0000 UTC m=+0.468140320 container attach 2f1420af7749dacf84f22abbcd24e391c6b1ccb0e521effb7d3769f16c3f9f2b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-56e81df, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-23T03:33:31.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 76/726 objects degraded (10.468%), 21 pgs degraded (PG_DEGRADED) 2024-04-23T03:33:31.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 76/726 objects degraded (10.468%), 21 pgs degraded (PG_DEGRADED) 2024-04-23T03:33:31.833 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:33:31.833 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:33:31.833 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 bash[67327]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-23T03:33:31.834 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-23T03:33:31.835 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-23T03:33:31.835 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-04-23T03:33:31.835 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-04-23T03:33:31.835 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate[67342]: --> ceph-volume raw activate successful for osd ID: 0 2024-04-23T03:33:31.835 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:31 smithi007 podman[67327]: 2024-04-23 03:33:31.603990897 +0000 UTC m=+0.905836893 container died 2f1420af7749dacf84f22abbcd24e391c6b1ccb0e521effb7d3769f16c3f9f2b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T03:33:31.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:31 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 76/726 objects degraded (10.468%), 21 pgs degraded (PG_DEGRADED) 2024-04-23T03:33:32.583 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 podman[67485]: 2024-04-23 03:33:32.237834278 +0000 UTC m=+0.628052702 container remove 2f1420af7749dacf84f22abbcd24e391c6b1ccb0e521effb7d3769f16c3f9f2b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:33:32.583 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 podman[67549]: 2024-04-23 03:33:32.356606112 +0000 UTC m=+0.018860991 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:33:32.583 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 podman[67549]: 2024-04-23 03:33:32.455654863 +0000 UTC m=+0.117909761 container create f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, CEPH_POINT_RELEASE=) 2024-04-23T03:33:32.866 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:32 smithi007 ceph-mon[64248]: pgmap v43: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:32.867 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:32 smithi007 ceph-mon[61772]: pgmap v43: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:32.867 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 podman[67549]: 2024-04-23 03:33:32.76657195 +0000 UTC m=+0.428826810 container init f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-56e81df, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416) 2024-04-23T03:33:32.867 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 podman[67549]: 2024-04-23 03:33:32.769569578 +0000 UTC m=+0.431824430 container start f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-04-23T03:33:32.867 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 ceph-osd[67568]: -- 172.21.15.7:0/882411198 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564fecb63680 con 0x564fecb2cc00 2024-04-23T03:33:32.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:32 smithi149 ceph-mon[54848]: pgmap v43: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:33.315 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 bash[67549]: f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f 2024-04-23T03:33:33.315 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:32 smithi007 systemd[1]: Started Ceph osd.0 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:33:33.582 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:33 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:33:33.313+0000 7f685f7f67c0 -1 Falling back to public interface 2024-04-23T03:33:33.918 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:34.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:33 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:34.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:33 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:34.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:33 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:34.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:33 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:33 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:33 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:35.029 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:33:35.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:34 smithi007 ceph-mon[64248]: pgmap v44: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:35.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:34 smithi007 ceph-mon[61772]: pgmap v44: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:34 smithi149 ceph-mon[54848]: pgmap v44: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:35.743 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:33:35.743 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (2m) 59s ago 7m 15.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:33:35.743 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (2m) 45s ago 7m 47.6M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:33:35.743 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (114s) 59s ago 6m 49.5M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:33:35.743 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (105s) 45s ago 11m 450M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:33:35.743 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (2m) 59s ago 13m 493M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (90s) 59s ago 13m 40.8M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (50s) 45s ago 12m 18.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (64s) 59s ago 12m 18.3M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (2m) 59s ago 8m 17.8M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (2m) 45s ago 8m 17.0M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 starting - - - 2737M 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (11m) 59s ago 11m 56.4M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (10m) 59s ago 10m 49.5M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (10m) 59s ago 10m 53.2M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (10m) 45s ago 10m 56.4M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:33:35.744 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (9m) 45s ago 9m 51.8M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (9m) 45s ago 9m 52.7M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (8m) 45s ago 8m 55.9M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (111s) 45s ago 8m 60.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (6m) 59s ago 6m 73.6M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (6m) 45s ago 6m 75.9M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (6m) 59s ago 6m 75.2M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:33:35.745 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (6m) 45s ago 6m 75.1M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:33:36.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:35 smithi007 ceph-mon[61772]: from='client.44211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:36.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:35 smithi007 ceph-mon[64248]: from='client.44211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:33:36.188 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 5 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:33:36.189 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:33:36.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:33:36] "GET /metrics HTTP/1.1" 200 38247 "" "Prometheus/2.33.4" 2024-04-23T03:33:36.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:33:36.050+0000 7f685f7f67c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-04-23T03:33:36.333 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:33:36.080+0000 7f685f7f67c0 -1 osd.0 105 log_to_monitors true 2024-04-23T03:33:36.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:35 smithi149 ceph-mon[54848]: from='client.44211 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:33:36.545 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:33:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:33:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "5/23 daemons upgraded", 2024-04-23T03:33:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:33:36.546 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:33:36.546 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 76/726 objects degraded (10.468%), 21 pgs degraded 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: osd.0 (root=default,host=smithi007) is down 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 76/726 objects degraded (10.468%), 21 pgs degraded 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 1.0 is active+undersized+degraded, acting [7,6] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.0 is active+undersized+degraded, acting [7,1] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.4 is active+undersized+degraded, acting [1,7] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.f is active+undersized+degraded, acting [4,2] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.19 is active+undersized+degraded, acting [3,6] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.1e is active+undersized+degraded, acting [3,5] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1 is active+undersized+degraded, acting [4,3] 2024-04-23T03:33:37.001 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.3 is active+undersized+degraded, acting [4,6] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.6 is active+undersized+degraded, acting [1,4] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.7 is active+undersized+degraded, acting [3,7] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.b is active+undersized+degraded, acting [3,4] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.f is active+undersized+degraded, acting [7,4] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.10 is active+undersized+degraded, acting [6,5] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.12 is active+undersized+degraded, acting [7,3] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.17 is active+undersized+degraded, acting [5,3] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.18 is active+undersized+degraded, acting [3,1] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1b is active+undersized+degraded, acting [4,7] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1f is active+undersized+degraded, acting [5,2] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.0 is active+undersized+degraded, acting [3,7] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.3 is active+undersized+degraded, acting [5,7] 2024-04-23T03:33:37.002 INFO:teuthology.orchestra.run.smithi007.stdout: pg 5.1d is active+undersized+degraded, acting [1,4] 2024-04-23T03:33:37.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[64248]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:37.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[64248]: pgmap v45: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:37.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[64248]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:37.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[64248]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:33:37.206 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3396830180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:37.206 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:33:36.992+0000 7f6850c4e700 -1 osd.0 105 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:33:37.207 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[61772]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:37.207 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[61772]: pgmap v45: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:37.207 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[61772]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:37.207 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[61772]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:33:37.207 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:36 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3396830180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:36 smithi149 ceph-mon[54848]: from='client.44217 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:36 smithi149 ceph-mon[54848]: pgmap v45: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:36 smithi149 ceph-mon[54848]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:37.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:36 smithi149 ceph-mon[54848]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-23T03:33:37.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:36 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3396830180' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:38.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:38.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='client.44238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:38.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T03:33:38.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: osdmap e108: 8 total, 7 up, 8 in 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2044337887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='client.44238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: osdmap e108: 8 total, 7 up, 8 in 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2044337887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='client.44238 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: osdmap e108: 8 total, 7 up, 8 in 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2044337887' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:38.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-23T03:33:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:39.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: pgmap v47: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:39.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:33:39.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296] boot 2024-04-23T03:33:39.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: osdmap e109: 8 total, 8 up, 8 in 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: pgmap v47: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296] boot 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: osdmap e109: 8 total, 8 up, 8 in 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:39.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:38 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: pgmap v47: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 76/726 objects degraded (10.468%) 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: osd.0 [v2:172.21.15.7:6802/2136361296,v1:172.21.15.7:6803/2136361296] boot 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: osdmap e109: 8 total, 8 up, 8 in 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:38 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:40.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:39 smithi007 ceph-mon[61772]: osdmap e110: 8 total, 8 up, 8 in 2024-04-23T03:33:40.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:39 smithi007 ceph-mon[64248]: osdmap e110: 8 total, 8 up, 8 in 2024-04-23T03:33:40.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:39 smithi149 ceph-mon[54848]: osdmap e110: 8 total, 8 up, 8 in 2024-04-23T03:33:41.323 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:40 smithi149 ceph-mon[54848]: pgmap v50: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/726 objects degraded (10.468%) 2024-04-23T03:33:41.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:40 smithi007 ceph-mon[64248]: pgmap v50: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/726 objects degraded (10.468%) 2024-04-23T03:33:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:40 smithi007 ceph-mon[61772]: pgmap v50: 161 pgs: 21 active+undersized+degraded, 37 active+undersized, 103 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 76/726 objects degraded (10.468%) 2024-04-23T03:33:42.285 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:42 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:33:42.286 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:42 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:33:42.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:42 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:33:42.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:42 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:33:42.283+0000 7f1c772de700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-23T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: pgmap v51: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/726 objects degraded (4.408%) 2024-04-23T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:43.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:43.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-mon[54848]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-23T03:33:43.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: pgmap v51: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/726 objects degraded (4.408%) 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[64248]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: pgmap v51: 161 pgs: 7 active+undersized+degraded, 14 active+undersized, 140 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 32/726 objects degraded (4.408%) 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:33:43.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:33:43.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:43.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:43.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:43 smithi007 ceph-mon[61772]: Upgrade: unsafe to stop osd(s) at this time (6 PGs are or would become offline) 2024-04-23T03:33:43.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:44.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:44 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded) 2024-04-23T03:33:44.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:44 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:33:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:44 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded) 2024-04-23T03:33:44.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:44 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:33:44.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:44 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 32/726 objects degraded (4.408%), 7 pgs degraded) 2024-04-23T03:33:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:44 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:33:45.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:45 smithi007 ceph-mon[61772]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-23T03:33:45.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:45 smithi007 ceph-mon[64248]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-23T03:33:45.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:45 smithi149 ceph-mon[54848]: pgmap v52: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 782 B/s rd, 0 op/s 2024-04-23T03:33:46.528 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:33:46] "GET /metrics HTTP/1.1" 200 38329 "" "Prometheus/2.33.4" 2024-04-23T03:33:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-mon[61772]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:33:46.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:46.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-mon[64248]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:46 smithi149 ceph-mon[54848]: pgmap v53: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:33:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:33:47.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:47 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:47.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:47 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:47.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:47 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:48.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:48 smithi007 ceph-mon[64248]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:33:48.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:48 smithi007 ceph-mon[61772]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:33:48.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:48 smithi149 ceph-mon[54848]: pgmap v54: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:33:48.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:50.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:50 smithi007 ceph-mon[64248]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-23T03:33:50.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:50 smithi007 ceph-mon[61772]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-23T03:33:50.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:50 smithi149 ceph-mon[54848]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-23T03:33:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:51 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:51.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:51 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:51.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:51 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:33:52.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:52 smithi007 ceph-mon[64248]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:52.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:52 smithi007 ceph-mon[61772]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:52.857 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:52 smithi149 ceph-mon[54848]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:53.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:33:54.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:54 smithi007 ceph-mon[64248]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:54.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:54 smithi007 ceph-mon[61772]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:54.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:54 smithi149 ceph-mon[54848]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:56.534 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:33:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:33:56] "GET /metrics HTTP/1.1" 200 38329 "" "Prometheus/2.33.4" 2024-04-23T03:33:56.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:56 smithi007 ceph-mon[64248]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:56.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:56 smithi007 ceph-mon[61772]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:56.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:56 smithi149 ceph-mon[54848]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:33:57.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:57.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:57.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:57.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[64248]: Upgrade: osd.1 is safe to restart 2024-04-23T03:33:57.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:57.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:57.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:57.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:57 smithi007 ceph-mon[61772]: Upgrade: osd.1 is safe to restart 2024-04-23T03:33:57.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:57 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:33:57.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:57.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:57 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-04-23T03:33:57.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:57 smithi149 ceph-mon[54848]: Upgrade: osd.1 is safe to restart 2024-04-23T03:33:58.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:58 smithi007 ceph-mon[61772]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:58.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:58 smithi007 ceph-mon[64248]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:58.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:58 smithi149 ceph-mon[54848]: pgmap v59: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:33:58.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:33:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:33:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:00.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[64248]: Upgrade: Updating osd.1 2024-04-23T03:34:00.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[64248]: Deploying daemon osd.1 on smithi007 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[61772]: Upgrade: Updating osd.1 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:00.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:33:59 smithi007 ceph-mon[61772]: Deploying daemon osd.1 on smithi007 2024-04-23T03:34:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:59 smithi149 ceph-mon[54848]: Upgrade: Updating osd.1 2024-04-23T03:34:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:59 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:59 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-23T03:34:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:59 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:33:59 smithi149 ceph-mon[54848]: Deploying daemon osd.1 on smithi007 2024-04-23T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:00 smithi149 ceph-mon[54848]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:01.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:00 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:01.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:00 smithi007 ceph-mon[64248]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:01.194 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:00 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:01.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:00 smithi007 ceph-mon[61772]: pgmap v60: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:01.194 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:00 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:02.462 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:02 smithi007 systemd[1]: Stopping Ceph osd.1 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:34:02.832 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[32501]: 2024-04-23T03:34:02.467+0000 7fa55d83f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:34:02.832 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[32501]: 2024-04-23T03:34:02.467+0000 7fa55d83f700 -1 osd.1 110 *** Got signal Terminated *** 2024-04-23T03:34:02.833 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[32501]: 2024-04-23T03:34:02.467+0000 7fa55d83f700 -1 osd.1 110 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:34:03.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:02 smithi149 ceph-mon[54848]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:03.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:02 smithi149 ceph-mon[54848]: osd.1 marked itself down and dead 2024-04-23T03:34:03.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-mon[61772]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:03.318 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-mon[61772]: osd.1 marked itself down and dead 2024-04-23T03:34:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-mon[64248]: pgmap v61: 161 pgs: 161 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:03.318 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:02 smithi007 ceph-mon[64248]: osd.1 marked itself down and dead 2024-04-23T03:34:03.582 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:03 smithi007 podman[69571]: 2024-04-23 03:34:03.317212236 +0000 UTC m=+0.961507326 container died 0608886ea6229bfd3464295e94b0c63a6d2799c2a0cc86cf69c8e35f0d9e524b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, ceph=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-23T03:34:03.876 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:03 smithi007 podman[69571]: 2024-04-23 03:34:03.784640437 +0000 UTC m=+1.428935515 container cleanup 0608886ea6229bfd3464295e94b0c63a6d2799c2a0cc86cf69c8e35f0d9e524b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, ceph=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., release=754, vcs-type=git, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_CLEAN=True, version=8, io.openshift.tags=base centos centos-stream, distribution-scope=public, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8) 2024-04-23T03:34:03.876 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:03 smithi007 bash[69571]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1 2024-04-23T03:34:03.877 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:04.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:03 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:04.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:03 smithi149 ceph-mon[54848]: osdmap e111: 8 total, 7 up, 8 in 2024-04-23T03:34:04.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:03 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:04.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:03 smithi007 ceph-mon[64248]: osdmap e111: 8 total, 7 up, 8 in 2024-04-23T03:34:04.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:03 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:04.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:03 smithi007 ceph-mon[61772]: osdmap e111: 8 total, 7 up, 8 in 2024-04-23T03:34:04.333 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:04 smithi007 podman[69589]: 2024-04-23 03:34:04.077924555 +0000 UTC m=+0.762355790 container remove 0608886ea6229bfd3464295e94b0c63a6d2799c2a0cc86cf69c8e35f0d9e524b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux ) 2024-04-23T03:34:04.707 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:04 smithi007 podman[69679]: 2024-04-23 03:34:04.331101773 +0000 UTC m=+0.018360260 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:04.707 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:04 smithi007 podman[69679]: 2024-04-23 03:34:04.44558263 +0000 UTC m=+0.132841096 container create c58e12da13300619d78dd146d244231e0e229fdb8d3eaa1bd5a62feb2bd5ee2f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T03:34:05.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:04 smithi007 ceph-mon[61772]: pgmap v63: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:34:05.023 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:04 smithi007 ceph-mon[61772]: osdmap e112: 8 total, 7 up, 8 in 2024-04-23T03:34:05.024 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:04 smithi007 ceph-mon[64248]: pgmap v63: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:34:05.024 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:04 smithi007 ceph-mon[64248]: osdmap e112: 8 total, 7 up, 8 in 2024-04-23T03:34:05.024 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:04 smithi007 podman[69679]: 2024-04-23 03:34:04.70676679 +0000 UTC m=+0.394025257 container init c58e12da13300619d78dd146d244231e0e229fdb8d3eaa1bd5a62feb2bd5ee2f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, io.buildah.version=1.33.5) 2024-04-23T03:34:05.024 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:04 smithi007 podman[69679]: 2024-04-23 03:34:04.710448431 +0000 UTC m=+0.397706907 container start c58e12da13300619d78dd146d244231e0e229fdb8d3eaa1bd5a62feb2bd5ee2f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-23T03:34:05.024 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:04 smithi007 podman[69679]: 2024-04-23 03:34:04.790093195 +0000 UTC m=+0.477351671 container attach c58e12da13300619d78dd146d244231e0e229fdb8d3eaa1bd5a62feb2bd5ee2f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=main-56e81df, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T03:34:05.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:04 smithi149 ceph-mon[54848]: pgmap v63: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:34:05.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:04 smithi149 ceph-mon[54848]: osdmap e112: 8 total, 7 up, 8 in 2024-04-23T03:34:05.332 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:05 smithi007 podman[69679]: 2024-04-23 03:34:05.023732576 +0000 UTC m=+0.710991057 container died c58e12da13300619d78dd146d244231e0e229fdb8d3eaa1bd5a62feb2bd5ee2f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-23T03:34:06.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:05 smithi007 podman[69704]: 2024-04-23 03:34:05.74454817 +0000 UTC m=+0.715360411 container remove c58e12da13300619d78dd146d244231e0e229fdb8d3eaa1bd5a62feb2bd5ee2f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-23T03:34:06.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:05 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.1.service: Deactivated successfully. 2024-04-23T03:34:06.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:05 smithi007 systemd[1]: Stopped Ceph osd.1 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:34:06.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:05 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.1.service: Consumed 8.775s CPU time. 2024-04-23T03:34:06.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:05 smithi007 systemd[1]: Starting Ceph osd.1 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:34:06.350 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:34:06] "GET /metrics HTTP/1.1" 200 38328 "" "Prometheus/2.33.4" 2024-04-23T03:34:06.350 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 podman[69809]: 2024-04-23 03:34:06.203833941 +0000 UTC m=+0.109231666 container create 9de37e52e8265c89e2b223784fd496c1ab4b32aa81dd2db1d2b1454c278561a2 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:34:06.351 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 podman[69809]: 2024-04-23 03:34:06.113334697 +0000 UTC m=+0.018732379 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:06.832 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 podman[69809]: 2024-04-23 03:34:06.473130876 +0000 UTC m=+0.378528556 container init 9de37e52e8265c89e2b223784fd496c1ab4b32aa81dd2db1d2b1454c278561a2 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True) 2024-04-23T03:34:06.833 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 podman[69809]: 2024-04-23 03:34:06.476375604 +0000 UTC m=+0.381773286 container start 9de37e52e8265c89e2b223784fd496c1ab4b32aa81dd2db1d2b1454c278561a2 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate, ceph=True, org.label-schema.build-date=20240416, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-23T03:34:06.833 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 podman[69809]: 2024-04-23 03:34:06.556311953 +0000 UTC m=+0.461709643 container attach 9de37e52e8265c89e2b223784fd496c1ab4b32aa81dd2db1d2b1454c278561a2 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:34:07.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:06 smithi149 ceph-mon[54848]: pgmap v65: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:07.276 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-mon[64248]: pgmap v65: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:07.277 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-mon[61772]: pgmap v65: 161 pgs: 20 stale+active+clean, 141 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:07.277 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:07.277 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 bash[69809]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-23T03:34:07.278 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate[69823]: --> ceph-volume raw activate successful for osd ID: 1 2024-04-23T03:34:07.279 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:06 smithi007 podman[69809]: 2024-04-23 03:34:06.993157577 +0000 UTC m=+0.898555270 container died 9de37e52e8265c89e2b223784fd496c1ab4b32aa81dd2db1d2b1454c278561a2 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate, GIT_CLEAN=True, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:34:07.369 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:34:07.947 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:07 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:07.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:07 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:07.947 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:07 smithi007 podman[69956]: 2024-04-23 03:34:07.677836444 +0000 UTC m=+0.679449711 container remove 9de37e52e8265c89e2b223784fd496c1ab4b32aa81dd2db1d2b1454c278561a2 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-activate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:34:07.947 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:07 smithi007 podman[70071]: 2024-04-23 03:34:07.797348681 +0000 UTC m=+0.019284986 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:07.947 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:07 smithi007 podman[70071]: 2024-04-23 03:34:07.946640581 +0000 UTC m=+0.168576869 container create 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 30s ago 8m 23.8M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (2m) 78s ago 7m 47.6M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (2m) 30s ago 6m 52.4M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (2m) 78s ago 12m 450M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (2m) 30s ago 14m 502M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (2m) 30s ago 14m 42.3M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:34:08.070 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (82s) 78s ago 12m 18.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (97s) 30s ago 12m 36.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (3m) 30s ago 8m 17.7M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (3m) 78s ago 8m 17.0M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (35s) 30s ago 12m 12.7M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (11m) 30s ago 11m 57.5M 2737M 17.2.0 e1d6a67b021e 0608886ea622 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (11m) 30s ago 11m 50.8M 2737M 17.2.0 e1d6a67b021e ea3883b9717d 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (11m) 30s ago 11m 54.4M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (10m) 78s ago 10m 56.4M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (10m) 78s ago 10m 51.8M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (9m) 78s ago 9m 52.7M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (9m) 78s ago 9m 55.9M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (2m) 78s ago 8m 60.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (7m) 30s ago 7m 74.0M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:34:08.071 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (7m) 78s ago 7m 75.9M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:34:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (7m) 30s ago 7m 75.5M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:34:08.072 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (7m) 78s ago 7m 75.1M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:34:08.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:07 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:08.206 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:08 smithi007 podman[70071]: 2024-04-23 03:34:08.181687975 +0000 UTC m=+0.403624285 container init 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T03:34:08.206 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:08 smithi007 podman[70071]: 2024-04-23 03:34:08.185238856 +0000 UTC m=+0.407175154 container start 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-23T03:34:08.505 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:34:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:34:08.505 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 1 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:34:08.506 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:34:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:34:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-04-23T03:34:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 6 2024-04-23T03:34:08.507 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:34:08.507 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:34:08.507 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-osd[70132]: -- 172.21.15.7:0/3072971879 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b7e567d680 con 0x55b7e5647000 2024-04-23T03:34:08.507 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:08 smithi007 bash[70071]: 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 2024-04-23T03:34:08.507 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:08 smithi007 systemd[1]: Started Ceph osd.1 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:34:08.832 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:34:08.731+0000 7fa6aaced7c0 -1 Falling back to public interface 2024-04-23T03:34:08.869 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:34:08.869 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:34:08.869 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:34:08.869 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:34:08.869 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:34:08.869 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:34:08.870 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:34:08.870 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:34:08.870 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "6/23 daemons upgraded", 2024-04-23T03:34:08.870 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:34:08.870 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:34:08.870 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:34:08.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: pgmap v66: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:08.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:08.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:08.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:08 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2963614905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:09.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:09.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: pgmap v66: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:09.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:09.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:09.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:09.247 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2963614905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: from='client.54181 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: pgmap v66: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: from='client.54187 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: from='client.44262 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:09.248 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:09.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:09.249 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:08 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2963614905' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] OSD_DOWN: 1 osds down 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: osd.1 (root=default,host=smithi007) is down 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.0 is active+undersized+degraded, acting [7,0] 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-04-23T03:34:09.308 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.a is active+undersized+degraded, acting [6,4] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.e is active+undersized+degraded, acting [7,4] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.16 is active+undersized+degraded, acting [5,7] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1a is active+undersized+degraded, acting [4,2] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 3.1c is active+undersized+degraded, acting [5,4] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.9 is active+undersized+degraded, acting [4,3] 2024-04-23T03:34:09.309 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.d is active+undersized+degraded, acting [4,2] 2024-04-23T03:34:09.310 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.e is active+undersized+degraded, acting [4,6] 2024-04-23T03:34:09.310 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-04-23T03:34:09.310 INFO:teuthology.orchestra.run.smithi007.stdout: pg 4.1f is active+undersized+degraded, acting [6,5] 2024-04-23T03:34:09.310 INFO:teuthology.orchestra.run.smithi007.stdout: pg 5.1d is active+undersized+degraded, acting [4,0] 2024-04-23T03:34:09.310 INFO:teuthology.orchestra.run.smithi007.stdout: pg 6.1a is active+undersized+degraded, acting [4,5] 2024-04-23T03:34:10.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:09 smithi149 ceph-mon[54848]: from='client.54208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:10.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:09 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/564507838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:34:10.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:09 smithi007 ceph-mon[64248]: from='client.54208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:10.188 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:09 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/564507838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:34:10.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:09 smithi007 ceph-mon[61772]: from='client.54208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:10.189 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:09 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/564507838' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:34:11.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:10 smithi149 ceph-mon[54848]: pgmap v67: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:11.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:10 smithi007 ceph-mon[64248]: pgmap v67: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:11.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:10 smithi007 ceph-mon[61772]: pgmap v67: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:11.746 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:34:11.450+0000 7fa6aaced7c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-04-23T03:34:11.746 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:34:11.483+0000 7fa6aaced7c0 -1 osd.1 110 log_to_monitors true 2024-04-23T03:34:12.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:11 smithi007 ceph-mon[64248]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T03:34:12.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:11 smithi007 ceph-mon[61772]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T03:34:12.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:34:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:34:11.927+0000 7fa69c145700 -1 osd.1 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:34:12.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:11 smithi149 ceph-mon[54848]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-23T03:34:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:12 smithi149 ceph-mon[54848]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:12 smithi149 ceph-mon[54848]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T03:34:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:12 smithi149 ceph-mon[54848]: osdmap e113: 8 total, 7 up, 8 in 2024-04-23T03:34:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:12 smithi149 ceph-mon[54848]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:12 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:12 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[64248]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[64248]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[64248]: osdmap e113: 8 total, 7 up, 8 in 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[64248]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[61772]: pgmap v68: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[61772]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[61772]: osdmap e113: 8 total, 7 up, 8 in 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[61772]: from='osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:13.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:12 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:13.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:34:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[64248]: osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363] boot 2024-04-23T03:34:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[64248]: osdmap e114: 8 total, 8 up, 8 in 2024-04-23T03:34:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[64248]: osdmap e115: 8 total, 8 up, 8 in 2024-04-23T03:34:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:34:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[61772]: osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363] boot 2024-04-23T03:34:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[61772]: osdmap e114: 8 total, 8 up, 8 in 2024-04-23T03:34:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:13 smithi007 ceph-mon[61772]: osdmap e115: 8 total, 8 up, 8 in 2024-04-23T03:34:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:13 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-23T03:34:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:13 smithi149 ceph-mon[54848]: osd.1 [v2:172.21.15.7:6810/894240363,v1:172.21.15.7:6811/894240363] boot 2024-04-23T03:34:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:13 smithi149 ceph-mon[54848]: osdmap e114: 8 total, 8 up, 8 in 2024-04-23T03:34:14.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:13 smithi149 ceph-mon[54848]: osdmap e115: 8 total, 8 up, 8 in 2024-04-23T03:34:15.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:15 smithi007 ceph-mon[61772]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:15.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:15 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:15.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:15 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:15.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:15 smithi007 ceph-mon[64248]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:15.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:15 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:15.300 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:15 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:15.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:15 smithi149 ceph-mon[54848]: pgmap v71: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 101/726 objects degraded (13.912%) 2024-04-23T03:34:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:15 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:15.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:15 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:16.254 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:34:16] "GET /metrics HTTP/1.1" 200 38261 "" "Prometheus/2.33.4" 2024-04-23T03:34:16.786 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-mon[64248]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 101/726 objects degraded (13.912%) 2024-04-23T03:34:16.786 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:16.786 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:16.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-mon[61772]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 101/726 objects degraded (13.912%) 2024-04-23T03:34:16.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:16.786 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:16 smithi149 ceph-mon[54848]: pgmap v73: 161 pgs: 22 active+undersized+degraded, 41 active+undersized, 98 active+clean; 585 KiB data, 131 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 101/726 objects degraded (13.912%) 2024-04-23T03:34:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:17 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:17 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:17.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:17 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:17 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:17 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:17 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:17 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:18.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: Upgrade: osd.2 is safe to restart 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded) 2024-04-23T03:34:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: Upgrade: osd.2 is safe to restart 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded) 2024-04-23T03:34:18.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:18 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:34:18.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: pgmap v74: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: Upgrade: osd.2 is safe to restart 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 101/726 objects degraded (13.912%), 22 pgs degraded) 2024-04-23T03:34:18.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:18 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:34:20.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[64248]: Upgrade: Updating osd.2 2024-04-23T03:34:20.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:20.563 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[64248]: Deploying daemon osd.2 on smithi007 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[61772]: Upgrade: Updating osd.2 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:20.564 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:20 smithi007 ceph-mon[61772]: Deploying daemon osd.2 on smithi007 2024-04-23T03:34:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:20 smithi149 ceph-mon[54848]: Upgrade: Updating osd.2 2024-04-23T03:34:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:20 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:20 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-23T03:34:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:20 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:20 smithi149 ceph-mon[54848]: Deploying daemon osd.2 on smithi007 2024-04-23T03:34:21.560 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:21 smithi007 ceph-mon[64248]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-04-23T03:34:21.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:21 smithi007 ceph-mon[61772]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-04-23T03:34:21.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:21 smithi149 ceph-mon[54848]: pgmap v75: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 675 B/s rd, 0 op/s 2024-04-23T03:34:22.833 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:22 smithi007 systemd[1]: Stopping Ceph osd.2 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:34:23.238 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[35443]: 2024-04-23T03:34:22.957+0000 7f91a22d3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:34:23.238 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[35443]: 2024-04-23T03:34:22.957+0000 7f91a22d3700 -1 osd.2 115 *** Got signal Terminated *** 2024-04-23T03:34:23.238 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:22 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[35443]: 2024-04-23T03:34:22.957+0000 7f91a22d3700 -1 osd.2 115 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:34:23.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:23 smithi149 ceph-mon[54848]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:23.505 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:23 smithi149 ceph-mon[54848]: osd.2 marked itself down and dead 2024-04-23T03:34:23.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:23 smithi007 ceph-mon[64248]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:23.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:23 smithi007 ceph-mon[64248]: osd.2 marked itself down and dead 2024-04-23T03:34:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:23 smithi007 ceph-mon[61772]: pgmap v76: 161 pgs: 161 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:23 smithi007 ceph-mon[61772]: osd.2 marked itself down and dead 2024-04-23T03:34:23.924 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:23 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:24.018 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:23 smithi007 podman[72020]: 2024-04-23 03:34:23.735312121 +0000 UTC m=+1.042613315 container died ea3883b9717d35e9aec2889785cdd6f1c0227b55b0fcdccace49c4b1632f0eb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, release=754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-type=git) 2024-04-23T03:34:24.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:24 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:24.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:24 smithi007 ceph-mon[64248]: osdmap e116: 8 total, 7 up, 8 in 2024-04-23T03:34:24.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:24 smithi007 ceph-mon[64248]: osdmap e117: 8 total, 7 up, 8 in 2024-04-23T03:34:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:24 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:24 smithi007 ceph-mon[61772]: osdmap e116: 8 total, 7 up, 8 in 2024-04-23T03:34:24.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:24 smithi007 ceph-mon[61772]: osdmap e117: 8 total, 7 up, 8 in 2024-04-23T03:34:24.334 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 podman[72020]: 2024-04-23 03:34:24.099886593 +0000 UTC m=+1.407187796 container cleanup ea3883b9717d35e9aec2889785cdd6f1c0227b55b0fcdccace49c4b1632f0eb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, name=centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-23T03:34:24.334 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 bash[72020]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2 2024-04-23T03:34:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:24 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:24 smithi149 ceph-mon[54848]: osdmap e116: 8 total, 7 up, 8 in 2024-04-23T03:34:24.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:24 smithi149 ceph-mon[54848]: osdmap e117: 8 total, 7 up, 8 in 2024-04-23T03:34:24.723 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 podman[72038]: 2024-04-23 03:34:24.40164342 +0000 UTC m=+0.668258913 container remove ea3883b9717d35e9aec2889785cdd6f1c0227b55b0fcdccace49c4b1632f0eb0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, release=754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, architecture=x86_64, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git, distribution-scope=public, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-23T03:34:24.988 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 podman[72133]: 2024-04-23 03:34:24.626811951 +0000 UTC m=+0.018921493 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:24.989 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 podman[72133]: 2024-04-23 03:34:24.760902443 +0000 UTC m=+0.153011972 container create acc5b2d7b75455e1e3dcf82baa9d92a83de8dbddc8873d488a9dc89224ff2af5 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T03:34:25.248 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 podman[72133]: 2024-04-23 03:34:24.988283073 +0000 UTC m=+0.380392603 container init acc5b2d7b75455e1e3dcf82baa9d92a83de8dbddc8873d488a9dc89224ff2af5 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T03:34:25.248 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:24 smithi007 podman[72133]: 2024-04-23 03:34:24.991907421 +0000 UTC m=+0.384016952 container start acc5b2d7b75455e1e3dcf82baa9d92a83de8dbddc8873d488a9dc89224ff2af5 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:34:25.248 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:25 smithi007 podman[72133]: 2024-04-23 03:34:25.06326227 +0000 UTC m=+0.455371793 container attach acc5b2d7b75455e1e3dcf82baa9d92a83de8dbddc8873d488a9dc89224ff2af5 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-23T03:34:25.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:25 smithi007 ceph-mon[61772]: pgmap v78: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:34:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:25 smithi007 ceph-mon[64248]: pgmap v78: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:34:25.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:25 smithi007 podman[72133]: 2024-04-23 03:34:25.299174435 +0000 UTC m=+0.691283972 container died acc5b2d7b75455e1e3dcf82baa9d92a83de8dbddc8873d488a9dc89224ff2af5 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=main-56e81df, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T03:34:25.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:25 smithi149 ceph-mon[54848]: pgmap v78: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:34:26.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:34:26] "GET /metrics HTTP/1.1" 200 38261 "" "Prometheus/2.33.4" 2024-04-23T03:34:26.333 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 podman[72159]: 2024-04-23 03:34:26.008930735 +0000 UTC m=+0.704425823 container remove acc5b2d7b75455e1e3dcf82baa9d92a83de8dbddc8873d488a9dc89224ff2af5 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416) 2024-04-23T03:34:26.333 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.2.service: Deactivated successfully. 2024-04-23T03:34:26.333 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 systemd[1]: Stopped Ceph osd.2 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:34:26.333 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.2.service: Consumed 7.037s CPU time. 2024-04-23T03:34:26.334 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 systemd[1]: Starting Ceph osd.2 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:34:26.755 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 podman[72260]: 2024-04-23 03:34:26.460385712 +0000 UTC m=+0.109471855 container create 2c216bad750eebdee76a6c88bf85a6e342831c06ee16530b1eec82f9d28b3771 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-04-23T03:34:26.755 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 podman[72260]: 2024-04-23 03:34:26.369782798 +0000 UTC m=+0.018868950 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:26.755 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 podman[72260]: 2024-04-23 03:34:26.68050925 +0000 UTC m=+0.329595390 container init 2c216bad750eebdee76a6c88bf85a6e342831c06ee16530b1eec82f9d28b3771 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T03:34:26.755 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 podman[72260]: 2024-04-23 03:34:26.683917834 +0000 UTC m=+0.333003980 container start 2c216bad750eebdee76a6c88bf85a6e342831c06ee16530b1eec82f9d28b3771 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:34:26.755 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:26 smithi007 podman[72260]: 2024-04-23 03:34:26.754474745 +0000 UTC m=+0.403560889 container attach 2c216bad750eebdee76a6c88bf85a6e342831c06ee16530b1eec82f9d28b3771 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-23T03:34:27.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:26 smithi007 ceph-mon[64248]: pgmap v80: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:27.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:26 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:27.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:26 smithi007 ceph-mon[61772]: pgmap v80: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:27.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:26 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:27.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:26 smithi149 ceph-mon[54848]: pgmap v80: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:27.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:26 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 bash[72260]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T03:34:27.583 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-23T03:34:27.584 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-23T03:34:27.584 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-23T03:34:27.584 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-04-23T03:34:27.584 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-04-23T03:34:27.584 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate[72274]: --> ceph-volume raw activate successful for osd ID: 2 2024-04-23T03:34:27.584 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 podman[72260]: 2024-04-23 03:34:27.208698583 +0000 UTC m=+0.857784729 container died 2c216bad750eebdee76a6c88bf85a6e342831c06ee16530b1eec82f9d28b3771 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:34:28.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:28.050 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:27 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:28.050 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:27 smithi007 podman[72418]: 2024-04-23 03:34:27.82508433 +0000 UTC m=+0.611040005 container remove 2c216bad750eebdee76a6c88bf85a6e342831c06ee16530b1eec82f9d28b3771 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-04-23T03:34:28.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:27 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:28.303 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 podman[72482]: 2024-04-23 03:34:27.952300929 +0000 UTC m=+0.019172105 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:28.303 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 podman[72482]: 2024-04-23 03:34:28.075955745 +0000 UTC m=+0.142826901 container create 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS) 2024-04-23T03:34:28.582 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 podman[72482]: 2024-04-23 03:34:28.303514945 +0000 UTC m=+0.370386101 container init 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=main-56e81df, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-23T03:34:28.582 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 podman[72482]: 2024-04-23 03:34:28.30680566 +0000 UTC m=+0.373676825 container start 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df) 2024-04-23T03:34:28.582 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-osd[72501]: -- 172.21.15.7:0/890990112 <== mon.1 v2:172.21.15.7:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55de33621680 con 0x55de335eb000 2024-04-23T03:34:28.582 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 bash[72482]: 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 2024-04-23T03:34:28.582 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 systemd[1]: Started Ceph osd.2 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:34:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:28 smithi149 ceph-mon[54848]: pgmap v81: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:28 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 64/726 objects degraded (8.815%), 13 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:28 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:28.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:28 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:28.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:29.082 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:34:28.846+0000 7fa56209d7c0 -1 Falling back to public interface 2024-04-23T03:34:29.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[61772]: pgmap v81: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 64/726 objects degraded (8.815%), 13 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[64248]: pgmap v81: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 64/726 objects degraded (8.815%), 13 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:29.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:28 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:31.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:30 smithi007 ceph-mon[61772]: pgmap v82: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:31.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:30 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:31.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:30 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:31.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:30 smithi007 ceph-mon[64248]: pgmap v82: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:31.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:30 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:31.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:30 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:31.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:30 smithi149 ceph-mon[54848]: pgmap v82: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:31.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:30 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:31.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:30 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:31.827 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:34:31.565+0000 7fa56209d7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-04-23T03:34:31.827 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:34:31.587+0000 7fa56209d7c0 -1 osd.2 115 log_to_monitors true 2024-04-23T03:34:32.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:31 smithi007 ceph-mon[64248]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:34:32.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:31 smithi007 ceph-mon[61772]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:34:32.083 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:34:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:34:31.854+0000 7fa5534f5700 -1 osd.2 115 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:34:32.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:31 smithi149 ceph-mon[54848]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-23T03:34:33.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:32 smithi149 ceph-mon[54848]: pgmap v83: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:33.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:32 smithi149 ceph-mon[54848]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T03:34:33.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:32 smithi149 ceph-mon[54848]: osdmap e118: 8 total, 7 up, 8 in 2024-04-23T03:34:33.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:32 smithi149 ceph-mon[54848]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:33.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[64248]: pgmap v83: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:33.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[64248]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T03:34:33.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[64248]: osdmap e118: 8 total, 7 up, 8 in 2024-04-23T03:34:33.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[64248]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:33.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[61772]: pgmap v83: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 137 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:33.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[61772]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-23T03:34:33.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[61772]: osdmap e118: 8 total, 7 up, 8 in 2024-04-23T03:34:33.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:32 smithi007 ceph-mon[61772]: from='osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:33.841 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-mon[54848]: osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439] boot 2024-04-23T03:34:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-mon[54848]: osdmap e119: 8 total, 8 up, 8 in 2024-04-23T03:34:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:34:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:34.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:33 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:34.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:34.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[64248]: osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439] boot 2024-04-23T03:34:34.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[64248]: osdmap e119: 8 total, 8 up, 8 in 2024-04-23T03:34:34.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:34:34.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:34.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:34.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:34.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[61772]: osd.2 [v2:172.21.15.7:6818/250647439,v1:172.21.15.7:6819/250647439] boot 2024-04-23T03:34:34.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[61772]: osdmap e119: 8 total, 8 up, 8 in 2024-04-23T03:34:34.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-23T03:34:34.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:34.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:33 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:35.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[64248]: pgmap v86: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:35.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[64248]: osdmap e120: 8 total, 8 up, 8 in 2024-04-23T03:34:35.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:35.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[61772]: pgmap v86: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[61772]: osdmap e120: 8 total, 8 up, 8 in 2024-04-23T03:34:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:35.052 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:34 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:34 smithi149 ceph-mon[54848]: pgmap v86: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:34 smithi149 ceph-mon[54848]: osdmap e120: 8 total, 8 up, 8 in 2024-04-23T03:34:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:34 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:35.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:34 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:36.341 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:34:36] "GET /metrics HTTP/1.1" 200 38421 "" "Prometheus/2.33.4" 2024-04-23T03:34:37.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:36 smithi149 ceph-mon[54848]: pgmap v88: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:37.296 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:36 smithi007 ceph-mon[64248]: pgmap v88: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:37.297 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:36 smithi007 ceph-mon[61772]: pgmap v88: 161 pgs: 13 active+undersized+degraded, 34 active+undersized, 114 active+clean; 585 KiB data, 152 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 64/726 objects degraded (8.815%) 2024-04-23T03:34:38.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:37 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-23T03:34:38.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:38.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:38.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:38.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:38.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:38.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:38.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:37 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-23T03:34:38.662 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:38.660Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-23T03:34:38.921 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:38.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-mon[54848]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:38.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-23T03:34:38.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-mon[54848]: Upgrade: osd.3 is safe to restart 2024-04-23T03:34:38.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/726 objects degraded (8.815%), 13 pgs degraded) 2024-04-23T03:34:38.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:38 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[64248]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[64248]: Upgrade: osd.3 is safe to restart 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/726 objects degraded (8.815%), 13 pgs degraded) 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[61772]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[61772]: Upgrade: osd.3 is safe to restart 2024-04-23T03:34:39.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 64/726 objects degraded (8.815%), 13 pgs degraded) 2024-04-23T03:34:39.084 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:38 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:34:39.699 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:34:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:34:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 7s ago 8m 25.9M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:34:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (3m) 110s ago 8m 47.6M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:34:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (2m) 7s ago 7m 52.7M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:34:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (2m) 110s ago 13m 450M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:34:40.402 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (3m) 7s ago 14m 506M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (2m) 7s ago 14m 46.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (115s) 110s ago 13m 18.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (2m) 7s ago 13m 39.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (3m) 7s ago 9m 16.7M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (3m) 110s ago 9m 17.0M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (68s) 7s ago 12m 44.6M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (32s) 7s ago 12m 43.4M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (12s) 7s ago 11m 15.2M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (11m) 7s ago 11m 56.6M 2737M 17.2.0 e1d6a67b021e e95fd3e644ea 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (11m) 110s ago 11m 56.4M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:34:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (10m) 110s ago 10m 51.8M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (10m) 110s ago 10m 52.7M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (10m) 110s ago 10m 55.9M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (2m) 110s ago 9m 60.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (7m) 7s ago 7m 74.2M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (7m) 110s ago 7m 75.9M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (7m) 7s ago 7m 75.7M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:34:40.404 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (7m) 110s ago 7m 75.1M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:34:40.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: Upgrade: Updating osd.3 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: Deploying daemon osd.3 on smithi007 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[61772]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: Upgrade: Updating osd.3 2024-04-23T03:34:40.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:34:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: Deploying daemon osd.3 on smithi007 2024-04-23T03:34:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-04-23T03:34:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:40.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:40 smithi007 ceph-mon[64248]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: Upgrade: Updating osd.3 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: Deploying daemon osd.3 on smithi007 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 668 B/s rd, 0 op/s 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: from='client.44298 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:40.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:40 smithi149 ceph-mon[54848]: from='client.54235 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:40.822 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 8 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:34:40.823 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:34:41.177 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "8/23 daemons upgraded", 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:34:41.178 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:34:41.606 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:34:41.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:41 smithi149 ceph-mon[54848]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:41.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:41 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3991187995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:41.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:41 smithi149 ceph-mon[54848]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:41.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:41 smithi007 ceph-mon[64248]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:41.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:41 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3991187995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:41.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:41 smithi007 ceph-mon[64248]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:41.739 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:41 smithi007 ceph-mon[61772]: from='client.54241 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:41.739 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:41 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3991187995' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:41.739 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:41 smithi007 ceph-mon[61772]: from='client.44322 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:34:42.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:42 smithi149 ceph-mon[54848]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:42.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:42 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3239679988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:34:42.723 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-mon[64248]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:42.723 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3239679988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:34:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-mon[61772]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:34:42.723 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3239679988' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:34:42.992 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:42 smithi007 systemd[1]: Stopping Ceph osd.3 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:34:43.332 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[38426]: 2024-04-23T03:34:42.990+0000 7f4c19d7e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:34:43.332 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[38426]: 2024-04-23T03:34:42.990+0000 7f4c19d7e700 -1 osd.3 120 *** Got signal Terminated *** 2024-04-23T03:34:43.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:42 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[38426]: 2024-04-23T03:34:42.990+0000 7f4c19d7e700 -1 osd.3 120 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:34:43.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:43 smithi149 ceph-mon[54848]: osd.3 marked itself down and dead 2024-04-23T03:34:43.672 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:43.812 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:43 smithi007 ceph-mon[64248]: osd.3 marked itself down and dead 2024-04-23T03:34:43.812 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:43 smithi007 ceph-mon[61772]: osd.3 marked itself down and dead 2024-04-23T03:34:44.082 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:43 smithi007 podman[74500]: 2024-04-23 03:34:43.811667477 +0000 UTC m=+0.941993419 container died e95fd3e644eadae195d0fd0dfe3a4e28599028b9c108290a7472ae2ce8946dea (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, version=8, architecture=x86_64, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-23T03:34:44.419 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:44 smithi007 podman[74500]: 2024-04-23 03:34:44.215627486 +0000 UTC m=+1.345953422 container cleanup e95fd3e644eadae195d0fd0dfe3a4e28599028b9c108290a7472ae2ce8946dea (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, architecture=x86_64, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , distribution-scope=public, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream) 2024-04-23T03:34:44.419 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:44 smithi007 bash[74500]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3 2024-04-23T03:34:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:44 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:44 smithi149 ceph-mon[54848]: osdmap e121: 8 total, 7 up, 8 in 2024-04-23T03:34:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:44 smithi149 ceph-mon[54848]: pgmap v93: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:34:44.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:44 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:44.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:44 smithi007 ceph-mon[64248]: osdmap e121: 8 total, 7 up, 8 in 2024-04-23T03:34:44.776 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:44 smithi007 ceph-mon[64248]: pgmap v93: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:34:44.776 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:44 smithi007 podman[74517]: 2024-04-23 03:34:44.450204994 +0000 UTC m=+0.636349589 container remove e95fd3e644eadae195d0fd0dfe3a4e28599028b9c108290a7472ae2ce8946dea (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vcs-type=git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=HEAD, version=8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.expose-services=, release=754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-23T03:34:44.777 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:44 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:34:44.777 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:44 smithi007 ceph-mon[61772]: osdmap e121: 8 total, 7 up, 8 in 2024-04-23T03:34:44.777 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:44 smithi007 ceph-mon[61772]: pgmap v93: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:34:45.046 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:44 smithi007 podman[74610]: 2024-04-23 03:34:44.77603618 +0000 UTC m=+0.114626828 container create 4aa207e0069c138bec484d1f25bebe0287d6f96c557107150ee83363cca445a1 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T03:34:45.046 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:44 smithi007 podman[74610]: 2024-04-23 03:34:44.680098472 +0000 UTC m=+0.018689134 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:45.332 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:45 smithi007 podman[74610]: 2024-04-23 03:34:45.045654683 +0000 UTC m=+0.384245341 container init 4aa207e0069c138bec484d1f25bebe0287d6f96c557107150ee83363cca445a1 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:34:45.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:45 smithi007 podman[74610]: 2024-04-23 03:34:45.049301926 +0000 UTC m=+0.387892574 container start 4aa207e0069c138bec484d1f25bebe0287d6f96c557107150ee83363cca445a1 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:34:45.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:45 smithi007 podman[74610]: 2024-04-23 03:34:45.120053779 +0000 UTC m=+0.458644439 container attach 4aa207e0069c138bec484d1f25bebe0287d6f96c557107150ee83363cca445a1 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-23T03:34:45.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:45 smithi007 ceph-mon[64248]: osdmap e122: 8 total, 7 up, 8 in 2024-04-23T03:34:45.732 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:45 smithi007 podman[74610]: 2024-04-23 03:34:45.363696791 +0000 UTC m=+0.702287431 container died 4aa207e0069c138bec484d1f25bebe0287d6f96c557107150ee83363cca445a1 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , RELEASE=main-56e81df) 2024-04-23T03:34:45.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:45 smithi007 ceph-mon[61772]: osdmap e122: 8 total, 7 up, 8 in 2024-04-23T03:34:45.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:45 smithi149 ceph-mon[54848]: osdmap e122: 8 total, 7 up, 8 in 2024-04-23T03:34:46.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:34:46] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-04-23T03:34:46.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 podman[74636]: 2024-04-23 03:34:46.04113333 +0000 UTC m=+0.667490412 container remove 4aa207e0069c138bec484d1f25bebe0287d6f96c557107150ee83363cca445a1 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-04-23T03:34:46.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.3.service: Deactivated successfully. 2024-04-23T03:34:46.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 systemd[1]: Stopped Ceph osd.3 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:34:46.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.3.service: Consumed 8.842s CPU time. 2024-04-23T03:34:46.333 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 systemd[1]: Starting Ceph osd.3 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:34:46.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[64248]: pgmap v95: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:46.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:46.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:46.748 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:46.749 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 podman[74742]: 2024-04-23 03:34:46.399835354 +0000 UTC m=+0.018804610 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:46.749 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 podman[74742]: 2024-04-23 03:34:46.509105095 +0000 UTC m=+0.128074384 container create bcf3766abe2686a8f444a1791da957361f5b69cc3aa160d8a549218c008ff372 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD) 2024-04-23T03:34:46.749 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 podman[74742]: 2024-04-23 03:34:46.744918782 +0000 UTC m=+0.363888028 container init bcf3766abe2686a8f444a1791da957361f5b69cc3aa160d8a549218c008ff372 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=main-56e81df, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-23T03:34:46.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[61772]: pgmap v95: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:46.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:46.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:46.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:46 smithi149 ceph-mon[54848]: pgmap v95: 161 pgs: 27 stale+active+clean, 134 active+clean; 585 KiB data, 153 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-04-23T03:34:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:34:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:47.082 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 podman[74742]: 2024-04-23 03:34:46.748210156 +0000 UTC m=+0.367179402 container start bcf3766abe2686a8f444a1791da957361f5b69cc3aa160d8a549218c008ff372 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-04-23T03:34:47.082 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:46 smithi007 podman[74742]: 2024-04-23 03:34:46.853401872 +0000 UTC m=+0.472371126 container attach bcf3766abe2686a8f444a1791da957361f5b69cc3aa160d8a549218c008ff372 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 bash[74742]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-23T03:34:47.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate[74758]: --> ceph-volume raw activate successful for osd ID: 3 2024-04-23T03:34:47.543 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 podman[74742]: 2024-04-23 03:34:47.267193283 +0000 UTC m=+0.886162519 container died bcf3766abe2686a8f444a1791da957361f5b69cc3aa160d8a549218c008ff372 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, GIT_BRANCH=HEAD) 2024-04-23T03:34:47.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:47.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:47 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:47.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:47 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:48.240 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:47 smithi007 podman[74885]: 2024-04-23 03:34:47.907698713 +0000 UTC m=+0.629281415 container remove bcf3766abe2686a8f444a1791da957361f5b69cc3aa160d8a549218c008ff372 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-activate, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T03:34:48.240 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 podman[74948]: 2024-04-23 03:34:48.035143207 +0000 UTC m=+0.019295557 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:34:48.240 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 podman[74948]: 2024-04-23 03:34:48.133639943 +0000 UTC m=+0.117792275 container create e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T03:34:48.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[64248]: pgmap v96: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:48.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 99/726 objects degraded (13.636%), 27 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:48.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:48.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:48.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[61772]: pgmap v96: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:48.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 99/726 objects degraded (13.636%), 27 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:48.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:48.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:48.584 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 podman[74948]: 2024-04-23 03:34:48.353133043 +0000 UTC m=+0.337285375 container init e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T03:34:48.584 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 podman[74948]: 2024-04-23 03:34:48.356659016 +0000 UTC m=+0.340811349 container start e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-23T03:34:48.584 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-osd[74968]: -- 172.21.15.7:0/3904560991 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555d8aad3680 con 0x555d8aa9d000 2024-04-23T03:34:48.584 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 bash[74948]: e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c 2024-04-23T03:34:48.584 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 systemd[1]: Started Ceph osd.3 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:34:48.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:48 smithi149 ceph-mon[54848]: pgmap v96: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:48.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:48 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 99/726 objects degraded (13.636%), 27 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:48.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:48 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:48.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:48 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:48.923 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:48.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-23T03:34:48.923 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:49.332 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:48 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:34:48.895+0000 7f5e52b3c7c0 -1 Falling back to public interface 2024-04-23T03:34:50.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:50 smithi007 ceph-mon[64248]: pgmap v97: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:50.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:50 smithi007 ceph-mon[61772]: pgmap v97: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:50.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:50 smithi149 ceph-mon[54848]: pgmap v97: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:51.875 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:51 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:34:51.617+0000 7f5e52b3c7c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-04-23T03:34:51.875 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:51 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:34:51.653+0000 7f5e52b3c7c0 -1 osd.3 120 log_to_monitors true 2024-04-23T03:34:52.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-mon[64248]: pgmap v98: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:52.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-mon[64248]: from='osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:34:52.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-mon[64248]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:34:52.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-mon[61772]: pgmap v98: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:52.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-mon[61772]: from='osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:34:52.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-mon[61772]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:34:52.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:52 smithi149 ceph-mon[54848]: pgmap v98: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 154 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:52.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:52 smithi149 ceph-mon[54848]: from='osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:34:52.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:52 smithi149 ceph-mon[54848]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-23T03:34:53.332 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:34:52 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:34:52.838+0000 7f5e3ff8c700 -1 osd.3 120 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:34:53.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T03:34:53.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: osdmap e123: 8 total, 7 up, 8 in 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: from='osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: from='osd.3 ' entity='osd.3' 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: osdmap e123: 8 total, 7 up, 8 in 2024-04-23T03:34:53.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: from='osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:53.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:53.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: from='osd.3 ' entity='osd.3' 2024-04-23T03:34:53.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:53.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:53 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:53.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-23T03:34:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: osdmap e123: 8 total, 7 up, 8 in 2024-04-23T03:34:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: from='osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-23T03:34:53.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: from='osd.3 ' entity='osd.3' 2024-04-23T03:34:53.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:53.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:53 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:54.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: pgmap v100: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871] boot 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: osdmap e124: 8 total, 8 up, 8 in 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: pgmap v100: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871] boot 2024-04-23T03:34:54.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: osdmap e124: 8 total, 8 up, 8 in 2024-04-23T03:34:54.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:34:54.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:54.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:54 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:54.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: pgmap v100: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 99/726 objects degraded (13.636%) 2024-04-23T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: osd.3 [v2:172.21.15.7:6826/4248444871,v1:172.21.15.7:6827/4248444871] boot 2024-04-23T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: osdmap e124: 8 total, 8 up, 8 in 2024-04-23T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-23T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:54.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:54 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:56.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:55 smithi007 ceph-mon[64248]: osdmap e125: 8 total, 8 up, 8 in 2024-04-23T03:34:56.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:55 smithi007 ceph-mon[61772]: osdmap e125: 8 total, 8 up, 8 in 2024-04-23T03:34:56.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:55 smithi149 ceph-mon[54848]: osdmap e125: 8 total, 8 up, 8 in 2024-04-23T03:34:56.299 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:34:56] "GET /metrics HTTP/1.1" 200 38428 "" "Prometheus/2.33.4" 2024-04-23T03:34:57.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:56 smithi149 ceph-mon[54848]: pgmap v103: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 99/726 objects degraded (13.636%) 2024-04-23T03:34:57.203 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:56 smithi007 ceph-mon[64248]: pgmap v103: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 99/726 objects degraded (13.636%) 2024-04-23T03:34:57.203 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:56 smithi007 ceph-mon[61772]: pgmap v103: 161 pgs: 27 active+undersized+degraded, 39 active+undersized, 95 active+clean; 585 KiB data, 169 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 99/726 objects degraded (13.636%) 2024-04-23T03:34:57.833 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:34:57.717+0000 7f1c772de700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:58.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.173 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:57 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:34:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:34:58.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:34:58.335 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:57 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:34:58.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:58 smithi149 ceph-mon[54848]: pgmap v104: 161 pgs: 13 active+undersized+degraded, 14 active+undersized, 134 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 45/726 objects degraded (6.198%) 2024-04-23T03:34:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:58 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:34:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:58 smithi149 ceph-mon[54848]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-23T03:34:58.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:58 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:58.923 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:58.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-04-23T03:34:58.923 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:34:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:34:58.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:34:59.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[61772]: pgmap v104: 161 pgs: 13 active+undersized+degraded, 14 active+undersized, 134 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 45/726 objects degraded (6.198%) 2024-04-23T03:34:59.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:34:59.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[61772]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-23T03:34:59.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded (PG_DEGRADED) 2024-04-23T03:34:59.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[64248]: pgmap v104: 161 pgs: 13 active+undersized+degraded, 14 active+undersized, 134 active+clean; 585 KiB data, 170 MiB used, 715 GiB / 715 GiB avail; 45/726 objects degraded (6.198%) 2024-04-23T03:34:59.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:34:59.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[64248]: Upgrade: unsafe to stop osd(s) at this time (4 PGs are or would become offline) 2024-04-23T03:34:59.334 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:58 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:59 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded) 2024-04-23T03:35:00.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:34:59 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:35:00.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:59 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded) 2024-04-23T03:35:00.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:34:59 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:35:00.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:59 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/726 objects degraded (6.198%), 13 pgs degraded) 2024-04-23T03:35:00.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:34:59 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:35:01.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:00 smithi149 ceph-mon[54848]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:01.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:00 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:01.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:00 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:01.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:00 smithi007 ceph-mon[61772]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:01.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:00 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:01.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:00 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:00 smithi007 ceph-mon[64248]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:00 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:00 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:03.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:02 smithi149 ceph-mon[54848]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:03.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:02 smithi007 ceph-mon[61772]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:03.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:02 smithi007 ceph-mon[64248]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:03.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:05.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:04 smithi149 ceph-mon[54848]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:05.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:04 smithi007 ceph-mon[64248]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:05.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:04 smithi007 ceph-mon[61772]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:06.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:35:06] "GET /metrics HTTP/1.1" 200 38432 "" "Prometheus/2.33.4" 2024-04-23T03:35:07.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:06 smithi007 ceph-mon[61772]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:07.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:07.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:06 smithi007 ceph-mon[64248]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:07.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:07.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:06 smithi149 ceph-mon[54848]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:07.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:08.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:07 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:08.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:07 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:08.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:07 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:08.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:08 smithi149 ceph-mon[54848]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:08.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:09.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:08 smithi007 ceph-mon[64248]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:09.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:08 smithi007 ceph-mon[61772]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail 2024-04-23T03:35:11.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:10 smithi149 ceph-mon[54848]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-23T03:35:11.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:10 smithi007 ceph-mon[64248]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-23T03:35:11.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:10 smithi007 ceph-mon[61772]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 426 B/s rd, 0 op/s 2024-04-23T03:35:11.983 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (4m) 19s ago 9m 24.2M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (4m) 2m ago 8m 47.6M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:35:12.666 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (3m) 19s ago 8m 52.9M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (3m) 2m ago 13m 450M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (3m) 19s ago 15m 507M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (3m) 19s ago 15m 51.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (2m) 2m ago 13m 18.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (2m) 19s ago 14m 44.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 19s ago 9m 17.1M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (4m) 2m ago 9m 17.0M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (100s) 19s ago 13m 44.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (64s) 19s ago 12m 43.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (44s) 19s ago 12m 44.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:35:12.667 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (24s) 19s ago 12m 12.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (11m) 2m ago 11m 56.4M 1454M 17.2.0 e1d6a67b021e 101ff6e9805b 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (11m) 2m ago 11m 51.8M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (10m) 2m ago 10m 52.7M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (10m) 2m ago 10m 55.9M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (3m) 2m ago 9m 60.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (8m) 19s ago 8m 74.3M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (8m) 2m ago 8m 75.9M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (8m) 19s ago 8m 75.8M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:35:12.668 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (8m) 2m ago 8m 75.1M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:35:13.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[64248]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:13.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[64248]: from='client.44340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:13.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[64248]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:13.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:35:13.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[61772]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:13.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[61772]: from='client.44340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:13.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[61772]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:13.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:12 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-04-23T03:35:13.084 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 4 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 9 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:35:13.085 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:35:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:12 smithi149 ceph-mon[54848]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:12 smithi149 ceph-mon[54848]: from='client.44340 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:12 smithi149 ceph-mon[54848]: from='client.54277 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:13.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:12 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:35:13.428 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:35:13.428 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:35:13.428 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:35:13.428 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:35:13.428 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "9/23 daemons upgraded", 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:35:13.429 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:35:13.559 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:13.846 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:35:14.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:13 smithi149 ceph-mon[54848]: from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:14.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:13 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:35:14.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:13 smithi149 ceph-mon[54848]: Upgrade: osd.4 is safe to restart 2024-04-23T03:35:14.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:13 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3903165512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:14.163 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:13 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2391921329' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:35:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[64248]: from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:35:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[64248]: Upgrade: osd.4 is safe to restart 2024-04-23T03:35:14.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3903165512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:14.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2391921329' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:35:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[61772]: from='client.54283 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-04-23T03:35:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[61772]: Upgrade: osd.4 is safe to restart 2024-04-23T03:35:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3903165512' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:14.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:13 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2391921329' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:35:15.369 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:15.369 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:15.369 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: Upgrade: Updating osd.4 2024-04-23T03:35:15.370 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:15.370 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:35:15.370 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:15.370 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:15 smithi149 ceph-mon[54848]: Deploying daemon osd.4 on smithi149 2024-04-23T03:35:15.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:15.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:15.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: Upgrade: Updating osd.4 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[64248]: Deploying daemon osd.4 on smithi149 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: from='client.44364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: Upgrade: Updating osd.4 2024-04-23T03:35:15.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:15.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-23T03:35:15.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:15.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:15 smithi007 ceph-mon[61772]: Deploying daemon osd.4 on smithi149 2024-04-23T03:35:16.551 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:16.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:16.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:35:16] "GET /metrics HTTP/1.1" 200 38434 "" "Prometheus/2.33.4" 2024-04-23T03:35:17.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:17 smithi149 ceph-mon[54848]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:17.422 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:17 smithi149 systemd[1]: Stopping Ceph osd.4 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:35:17.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:17 smithi007 ceph-mon[64248]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:17.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:17 smithi007 ceph-mon[61772]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:17.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[26091]: 2024-04-23T03:35:17.430+0000 7f03d1121700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:35:17.923 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[26091]: 2024-04-23T03:35:17.430+0000 7f03d1121700 -1 osd.4 125 *** Got signal Terminated *** 2024-04-23T03:35:17.923 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[26091]: 2024-04-23T03:35:17.430+0000 7f03d1121700 -1 osd.4 125 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:35:18.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:18 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:18.514 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:18 smithi007 ceph-mon[64248]: osd.4 marked itself down and dead 2024-04-23T03:35:18.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:18 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:18.514 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:18 smithi007 ceph-mon[61772]: osd.4 marked itself down and dead 2024-04-23T03:35:18.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:18 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:18.661 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:18 smithi149 ceph-mon[54848]: osd.4 marked itself down and dead 2024-04-23T03:35:18.921 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:18 smithi149 podman[57566]: 2024-04-23 03:35:18.679258562 +0000 UTC m=+1.377299796 container died 101ff6e9805bdde9358b010d0df91af4f1052ba524ea7e66258af948111edf82 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-type=git, GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vendor=Red Hat, Inc., architecture=x86_64, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, ceph=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-23T03:35:18.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:18 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:19.199 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57566]: 2024-04-23 03:35:19.006112938 +0000 UTC m=+1.704154163 container cleanup 101ff6e9805bdde9358b010d0df91af4f1052ba524ea7e66258af948111edf82 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, release=754, vcs-type=git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, name=centos-stream, vendor=Red Hat, Inc.) 2024-04-23T03:35:19.199 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 bash[57566]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4 2024-04-23T03:35:19.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:19 smithi007 ceph-mon[61772]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:19.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:19 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:35:19.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:19 smithi007 ceph-mon[61772]: osdmap e126: 8 total, 7 up, 8 in 2024-04-23T03:35:19.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:19 smithi007 ceph-mon[64248]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:19.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:19 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:35:19.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:19 smithi007 ceph-mon[64248]: osdmap e126: 8 total, 7 up, 8 in 2024-04-23T03:35:19.625 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57582]: 2024-04-23 03:35:19.198992615 +0000 UTC m=+0.521945489 container remove 101ff6e9805bdde9358b010d0df91af4f1052ba524ea7e66258af948111edf82 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=HEAD, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8) 2024-04-23T03:35:19.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:19 smithi149 ceph-mon[54848]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:19.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:19 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:35:19.625 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:19 smithi149 ceph-mon[54848]: osdmap e126: 8 total, 7 up, 8 in 2024-04-23T03:35:19.894 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57673]: 2024-04-23 03:35:19.624957428 +0000 UTC m=+0.109427833 container create 338e3a577a47887fa99aeb1c1bc4fe7eca207d51f01249ffaca7fcb8159b18aa (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-23T03:35:19.894 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57673]: 2024-04-23 03:35:19.533879936 +0000 UTC m=+0.018350341 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:35:19.894 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57673]: 2024-04-23 03:35:19.819273372 +0000 UTC m=+0.303743776 container init 338e3a577a47887fa99aeb1c1bc4fe7eca207d51f01249ffaca7fcb8159b18aa (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-23T03:35:19.894 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57673]: 2024-04-23 03:35:19.822431785 +0000 UTC m=+0.306902192 container start 338e3a577a47887fa99aeb1c1bc4fe7eca207d51f01249ffaca7fcb8159b18aa (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True) 2024-04-23T03:35:19.895 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:19 smithi149 podman[57673]: 2024-04-23 03:35:19.89424903 +0000 UTC m=+0.378719440 container attach 338e3a577a47887fa99aeb1c1bc4fe7eca207d51f01249ffaca7fcb8159b18aa (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-23T03:35:20.172 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:20 smithi149 podman[57673]: 2024-04-23 03:35:20.115955002 +0000 UTC m=+0.600425407 container died 338e3a577a47887fa99aeb1c1bc4fe7eca207d51f01249ffaca7fcb8159b18aa (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T03:35:20.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:20 smithi007 ceph-mon[61772]: osdmap e127: 8 total, 7 up, 8 in 2024-04-23T03:35:20.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:20 smithi007 ceph-mon[64248]: osdmap e127: 8 total, 7 up, 8 in 2024-04-23T03:35:20.647 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:20 smithi149 ceph-mon[54848]: osdmap e127: 8 total, 7 up, 8 in 2024-04-23T03:35:20.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:20 smithi149 podman[57700]: 2024-04-23 03:35:20.647401839 +0000 UTC m=+0.522232247 container remove 338e3a577a47887fa99aeb1c1bc4fe7eca207d51f01249ffaca7fcb8159b18aa (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.license=GPLv2) 2024-04-23T03:35:20.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:20 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.4.service: Deactivated successfully. 2024-04-23T03:35:20.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:20 smithi149 systemd[1]: Stopped Ceph osd.4 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:35:20.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:20 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.4.service: Consumed 9.130s CPU time. 2024-04-23T03:35:20.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:20 smithi149 systemd[1]: Starting Ceph osd.4 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:35:21.359 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-mon[54848]: pgmap v117: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:21.359 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 podman[57803]: 2024-04-23 03:35:20.992877775 +0000 UTC m=+0.018999515 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:35:21.359 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 podman[57803]: 2024-04-23 03:35:21.106905297 +0000 UTC m=+0.133027066 container create 3dcf12cb7b51882e6b921169fa39634021746752aaaec77c9014d2d881942c2a (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:35:21.359 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 podman[57803]: 2024-04-23 03:35:21.28459644 +0000 UTC m=+0.310718160 container init 3dcf12cb7b51882e6b921169fa39634021746752aaaec77c9014d2d881942c2a (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T03:35:21.360 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 podman[57803]: 2024-04-23 03:35:21.288142237 +0000 UTC m=+0.314263953 container start 3dcf12cb7b51882e6b921169fa39634021746752aaaec77c9014d2d881942c2a (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD) 2024-04-23T03:35:21.360 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 podman[57803]: 2024-04-23 03:35:21.359176369 +0000 UTC m=+0.385298088 container attach 3dcf12cb7b51882e6b921169fa39634021746752aaaec77c9014d2d881942c2a (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T03:35:21.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:21 smithi007 ceph-mon[64248]: pgmap v117: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:21.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:21 smithi007 ceph-mon[61772]: pgmap v117: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-23T03:35:22.089 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 bash[57803]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate[57818]: --> ceph-volume raw activate successful for osd ID: 4 2024-04-23T03:35:22.090 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:21 smithi149 podman[57803]: 2024-04-23 03:35:21.784699451 +0000 UTC m=+0.810821159 container died 3dcf12cb7b51882e6b921169fa39634021746752aaaec77c9014d2d881942c2a (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:35:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:22 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 63/726 objects degraded (8.678%), 12 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:22.422 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 podman[57942]: 2024-04-23 03:35:22.338156515 +0000 UTC m=+0.548169440 container remove 3dcf12cb7b51882e6b921169fa39634021746752aaaec77c9014d2d881942c2a (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-56e81df, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T03:35:22.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:22 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 63/726 objects degraded (8.678%), 12 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:22.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:22 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 63/726 objects degraded (8.678%), 12 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:22.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 podman[58008]: 2024-04-23 03:35:22.4528051 +0000 UTC m=+0.018195673 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:35:22.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 podman[58008]: 2024-04-23 03:35:22.55576764 +0000 UTC m=+0.121158215 container create 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:35:22.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 podman[58008]: 2024-04-23 03:35:22.69938116 +0000 UTC m=+0.264771735 container init 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-23T03:35:22.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 podman[58008]: 2024-04-23 03:35:22.702575588 +0000 UTC m=+0.267966160 container start 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=main-56e81df) 2024-04-23T03:35:22.922 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 ceph-osd[58027]: -- 172.21.15.149:0/4185732069 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5630467ab680 con 0x563046775000 2024-04-23T03:35:22.923 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 bash[58008]: 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 2024-04-23T03:35:22.923 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:22 smithi149 systemd[1]: Started Ceph osd.4 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:35:23.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:23 smithi007 ceph-mon[64248]: pgmap v118: 161 pgs: 12 active+undersized+degraded, 14 stale+active+clean, 20 active+undersized, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 63/726 objects degraded (8.678%) 2024-04-23T03:35:23.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:23 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:23.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:23 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:23 smithi007 ceph-mon[61772]: pgmap v118: 161 pgs: 12 active+undersized+degraded, 14 stale+active+clean, 20 active+undersized, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 63/726 objects degraded (8.678%) 2024-04-23T03:35:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:23 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:23.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:23 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:23.663 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:23 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:35:23.253+0000 7f793f4217c0 -1 Falling back to public interface 2024-04-23T03:35:23.663 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:23 smithi149 ceph-mon[54848]: pgmap v118: 161 pgs: 12 active+undersized+degraded, 14 stale+active+clean, 20 active+undersized, 115 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 63/726 objects degraded (8.678%) 2024-04-23T03:35:23.663 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:23 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:23.663 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:23 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:23.664 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:23 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:25.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:25 smithi007 ceph-mon[64248]: pgmap v119: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:25.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:25 smithi007 ceph-mon[61772]: pgmap v119: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:25.669 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:25 smithi149 ceph-mon[54848]: pgmap v119: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:26.316 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:26 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:35:25.998+0000 7f793f4217c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-04-23T03:35:26.316 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:26 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:35:26.032+0000 7f793f4217c0 -1 osd.4 125 log_to_monitors true 2024-04-23T03:35:26.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:26 smithi007 ceph-mon[64248]: from='osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:35:26.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:26 smithi007 ceph-mon[64248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:35:26.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:26 smithi007 ceph-mon[61772]: from='osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:35:26.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:26 smithi007 ceph-mon[61772]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:35:26.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:35:26] "GET /metrics HTTP/1.1" 200 38434 "" "Prometheus/2.33.4" 2024-04-23T03:35:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:26 smithi149 ceph-mon[54848]: from='osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:35:26.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:26 smithi149 ceph-mon[54848]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-23T03:35:26.672 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:35:26 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:35:26.365+0000 7f7930879700 -1 osd.4 125 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:35:27.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: pgmap v120: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T03:35:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: osdmap e128: 8 total, 7 up, 8 in 2024-04-23T03:35:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: from='osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:35:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:35:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:27.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:27 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:27.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: pgmap v120: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: osdmap e128: 8 total, 7 up, 8 in 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: from='osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: pgmap v120: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 178 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:27.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-23T03:35:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: osdmap e128: 8 total, 7 up, 8 in 2024-04-23T03:35:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: from='osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:35:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:35:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:27.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:27 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859] boot 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: osdmap e129: 8 total, 8 up, 8 in 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: pgmap v123: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:28.615 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.616 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-mon[54848]: osdmap e130: 8 total, 8 up, 8 in 2024-04-23T03:35:28.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:28.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859] boot 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: osdmap e129: 8 total, 8 up, 8 in 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: pgmap v123: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[64248]: osdmap e130: 8 total, 8 up, 8 in 2024-04-23T03:35:28.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: osd.4 [v2:172.21.15.149:6800/707790859,v1:172.21.15.149:6801/707790859] boot 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: osdmap e129: 8 total, 8 up, 8 in 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: pgmap v123: 161 pgs: 25 active+undersized+degraded, 43 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 123/726 objects degraded (16.942%) 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:28.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:28 smithi007 ceph-mon[61772]: osdmap e130: 8 total, 8 up, 8 in 2024-04-23T03:35:28.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:29.657 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:29 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:30.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:29 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:30.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:29 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 118/726 objects degraded (16.253%), 24 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:30.798 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:30 smithi149 ceph-mon[54848]: pgmap v125: 161 pgs: 4 peering, 24 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 118/726 objects degraded (16.253%) 2024-04-23T03:35:30.798 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:30 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:30.798 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:30 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:31.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:30 smithi007 ceph-mon[64248]: pgmap v125: 161 pgs: 4 peering, 24 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 118/726 objects degraded (16.253%) 2024-04-23T03:35:31.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:30 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:31.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:30 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:31.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:30 smithi007 ceph-mon[61772]: pgmap v125: 161 pgs: 4 peering, 24 active+undersized+degraded, 40 active+undersized, 93 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 118/726 objects degraded (16.253%) 2024-04-23T03:35:31.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:30 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:31.083 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:30 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:31.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:31 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:35:31.321+0000 7f1c772de700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-04-23T03:35:32.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:32.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:35:32.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:32.423 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:32 smithi149 ceph-mon[54848]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-04-23T03:35:32.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:32.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[64248]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:32.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:32.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:32.585 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:32 smithi007 ceph-mon[61772]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-04-23T03:35:33.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:33 smithi149 ceph-mon[54848]: pgmap v126: 161 pgs: 4 peering, 5 active+undersized+degraded, 13 active+undersized, 139 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/726 objects degraded (3.994%) 2024-04-23T03:35:33.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:33 smithi007 ceph-mon[64248]: pgmap v126: 161 pgs: 4 peering, 5 active+undersized+degraded, 13 active+undersized, 139 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/726 objects degraded (3.994%) 2024-04-23T03:35:33.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:33 smithi007 ceph-mon[61772]: pgmap v126: 161 pgs: 4 peering, 5 active+undersized+degraded, 13 active+undersized, 139 active+clean; 585 KiB data, 189 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 29/726 objects degraded (3.994%) 2024-04-23T03:35:33.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:34.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:34 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/726 objects degraded (3.994%), 5 pgs degraded) 2024-04-23T03:35:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:34 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:35:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:34 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/726 objects degraded (3.994%), 5 pgs degraded) 2024-04-23T03:35:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:34 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:35:34.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:34 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 29/726 objects degraded (3.994%), 5 pgs degraded) 2024-04-23T03:35:34.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:34 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:35:35.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:35 smithi007 ceph-mon[64248]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-23T03:35:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:35 smithi007 ceph-mon[61772]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-23T03:35:35.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:35 smithi149 ceph-mon[54848]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-23T03:35:36.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:35:36] "GET /metrics HTTP/1.1" 200 38301 "" "Prometheus/2.33.4" 2024-04-23T03:35:37.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:37 smithi007 ceph-mon[64248]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:37 smithi007 ceph-mon[61772]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:37.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:37 smithi149 ceph-mon[54848]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:38.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:38 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:38.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:38 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:38.660 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:38 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:38.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:38 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:38.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:39.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:39 smithi007 ceph-mon[64248]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:35:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:39 smithi007 ceph-mon[61772]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:35:39.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:39 smithi149 ceph-mon[54848]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:35:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:41 smithi007 ceph-mon[64248]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-23T03:35:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:41 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:41 smithi007 ceph-mon[61772]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-23T03:35:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:41 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:41 smithi149 ceph-mon[54848]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-04-23T03:35:41.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:41 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:43.506 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:43 smithi149 ceph-mon[54848]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:43.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:43 smithi007 ceph-mon[64248]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:43.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:43 smithi007 ceph-mon[61772]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:43.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:44.229 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (4m) 51s ago 9m 24.2M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (4m) 18s ago 9m 54.8M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (4m) 51s ago 8m 52.9M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (3m) 18s ago 14m 452M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (4m) 51s ago 15m 507M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (3m) 51s ago 15m 51.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (2m) 18s ago 14m 34.8M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (3m) 51s ago 14m 44.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 51s ago 10m 17.1M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:35:44.916 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (4m) 18s ago 10m 19.5M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (2m) 51s ago 13m 44.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (97s) 51s ago 13m 43.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (76s) 51s ago 13m 44.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (56s) 51s ago 12m 12.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (22s) 18s ago 12m 12.7M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (11m) 18s ago 11m 56.1M 1454M 17.2.0 e1d6a67b021e bcd6034cd0a1 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (11m) 18s ago 11m 56.9M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (11m) 18s ago 11m 62.2M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (4m) 18s ago 10m 74.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (8m) 51s ago 8m 74.3M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (8m) 18s ago 8m 76.5M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (8m) 51s ago 8m 75.8M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:35:44.917 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (8m) 18s ago 8m 75.7M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 5 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:35:45.331 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:35:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:35:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-04-23T03:35:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 10 2024-04-23T03:35:45.332 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:35:45.332 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:35:45.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:45 smithi007 ceph-mon[64248]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:45.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:45 smithi007 ceph-mon[64248]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:45.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:45 smithi007 ceph-mon[61772]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:45.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:45 smithi007 ceph-mon[61772]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:45.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:45 smithi149 ceph-mon[54848]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:45.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:45 smithi149 ceph-mon[54848]: from='client.44382 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:45.687 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:35:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:35:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:35:45.687 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "10/23 daemons upgraded", 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:35:45.688 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:35:46.108 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:35:46.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:35:46] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.33.4" 2024-04-23T03:35:46.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[61772]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:46.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[61772]: from='client.44394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/587584403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/4290574248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[64248]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[64248]: from='client.44394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/587584403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:46.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:46 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/4290574248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:35:46.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:46 smithi149 ceph-mon[54848]: from='client.44388 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:46.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:46 smithi149 ceph-mon[54848]: from='client.44394 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:46.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:46 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/587584403' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:35:46.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:46.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:35:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:46 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/4290574248' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:35:47.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[61772]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:47.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[61772]: from='client.54337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:47.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:47.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:47.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[61772]: Upgrade: osd.5 is safe to restart 2024-04-23T03:35:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[64248]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[64248]: from='client.54337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:47.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:47 smithi007 ceph-mon[64248]: Upgrade: osd.5 is safe to restart 2024-04-23T03:35:47.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:47 smithi149 ceph-mon[54848]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:47.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:47 smithi149 ceph-mon[54848]: from='client.54337 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:35:47.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:47 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:47.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:47 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-04-23T03:35:47.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:47 smithi149 ceph-mon[54848]: Upgrade: osd.5 is safe to restart 2024-04-23T03:35:48.534 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:48 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:48.534 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:48 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:48.535 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:48 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:35:48.535 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:48 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:48.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:48.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:48.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:35:48.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:48.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:48.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:48.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-23T03:35:48.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:48 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:35:48.922 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:48.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"ceph_cluster\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.7\", device_class=\"ssd\", hostname=\"smithi007\", instance=\"172.21.15.149:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.7\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:49.511 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:49 smithi149 ceph-mon[54848]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:49.512 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:49 smithi149 ceph-mon[54848]: Upgrade: Updating osd.5 2024-04-23T03:35:49.512 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:49 smithi149 ceph-mon[54848]: Deploying daemon osd.5 on smithi149 2024-04-23T03:35:49.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:49 smithi007 ceph-mon[61772]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:49.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:49 smithi007 ceph-mon[61772]: Upgrade: Updating osd.5 2024-04-23T03:35:49.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:49 smithi007 ceph-mon[61772]: Deploying daemon osd.5 on smithi149 2024-04-23T03:35:49.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:49 smithi007 ceph-mon[64248]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:35:49.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:49 smithi007 ceph-mon[64248]: Upgrade: Updating osd.5 2024-04-23T03:35:49.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:49 smithi007 ceph-mon[64248]: Deploying daemon osd.5 on smithi149 2024-04-23T03:35:50.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:50 smithi149 systemd[1]: Stopping Ceph osd.5 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:35:51.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:51 smithi149 ceph-mon[54848]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:51.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:51 smithi149 ceph-mon[54848]: osd.5 marked itself down and dead 2024-04-23T03:35:51.422 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[29040]: 2024-04-23T03:35:51.090+0000 7f0dc813b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:35:51.422 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[29040]: 2024-04-23T03:35:51.090+0000 7f0dc813b700 -1 osd.5 130 *** Got signal Terminated *** 2024-04-23T03:35:51.422 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[29040]: 2024-04-23T03:35:51.090+0000 7f0dc813b700 -1 osd.5 130 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:35:51.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:51 smithi007 ceph-mon[64248]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:51.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:51 smithi007 ceph-mon[64248]: osd.5 marked itself down and dead 2024-04-23T03:35:51.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:51 smithi007 ceph-mon[61772]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:35:51.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:51 smithi007 ceph-mon[61772]: osd.5 marked itself down and dead 2024-04-23T03:35:52.050 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:51 smithi149 podman[59889]: 2024-04-23 03:35:51.765370225 +0000 UTC m=+0.774156757 container died bcd6034cd0a1b73969d6e936328d3a17e8516de6968afda594674265c722b75f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, release=754, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-type=git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-04-23T03:35:52.409 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:52 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:35:52.409 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:52 smithi149 ceph-mon[54848]: osdmap e131: 8 total, 7 up, 8 in 2024-04-23T03:35:52.409 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:52 smithi149 ceph-mon[54848]: osdmap e132: 8 total, 7 up, 8 in 2024-04-23T03:35:52.409 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 podman[59889]: 2024-04-23 03:35:52.159748527 +0000 UTC m=+1.168535041 container cleanup bcd6034cd0a1b73969d6e936328d3a17e8516de6968afda594674265c722b75f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, name=centos-stream, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, release=754, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-23T03:35:52.410 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 bash[59889]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5 2024-04-23T03:35:52.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:52 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:35:52.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:52 smithi007 ceph-mon[64248]: osdmap e131: 8 total, 7 up, 8 in 2024-04-23T03:35:52.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:52 smithi007 ceph-mon[64248]: osdmap e132: 8 total, 7 up, 8 in 2024-04-23T03:35:52.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:52 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:35:52.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:52 smithi007 ceph-mon[61772]: osdmap e131: 8 total, 7 up, 8 in 2024-04-23T03:35:52.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:52 smithi007 ceph-mon[61772]: osdmap e132: 8 total, 7 up, 8 in 2024-04-23T03:35:52.671 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 podman[59907]: 2024-04-23 03:35:52.408739362 +0000 UTC m=+0.642593431 container remove bcd6034cd0a1b73969d6e936328d3a17e8516de6968afda594674265c722b75f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=HEAD, release=754, maintainer=Guillaume Abrioux , ceph=True, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.component=centos-stream-container) 2024-04-23T03:35:53.028 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 podman[60002]: 2024-04-23 03:35:52.653562977 +0000 UTC m=+0.018561085 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:35:53.029 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 podman[60002]: 2024-04-23 03:35:52.767815859 +0000 UTC m=+0.132813936 container create 8ff96300f28cd102905468468fc67f73b942c92375a120bbbb31b9fa9155d345 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T03:35:53.029 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 podman[60002]: 2024-04-23 03:35:52.971038775 +0000 UTC m=+0.336036859 container init 8ff96300f28cd102905468468fc67f73b942c92375a120bbbb31b9fa9155d345 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, maintainer=Guillaume Abrioux ) 2024-04-23T03:35:53.029 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:52 smithi149 podman[60002]: 2024-04-23 03:35:52.974582377 +0000 UTC m=+0.339580453 container start 8ff96300f28cd102905468468fc67f73b942c92375a120bbbb31b9fa9155d345 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-23T03:35:53.029 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:53 smithi149 podman[60002]: 2024-04-23 03:35:53.028673483 +0000 UTC m=+0.393671564 container attach 8ff96300f28cd102905468468fc67f73b942c92375a120bbbb31b9fa9155d345 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-04-23T03:35:53.284 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:53 smithi149 podman[60002]: 2024-04-23 03:35:53.269894015 +0000 UTC m=+0.634892102 container died 8ff96300f28cd102905468468fc67f73b942c92375a120bbbb31b9fa9155d345 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=main-56e81df, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:35:53.557 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:53 smithi149 ceph-mon[54848]: pgmap v137: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:35:53.557 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:35:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:35:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi007\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"3d05926e-0120-11ef-bc93-c7b262605968\", domainname=\"(none)\", instance=\"smithi007\", job=\"node\", machine=\"x86_64\", nodename=\"smithi007\", release=\"5.14.0-437.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-04-23T03:35:53.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:53 smithi007 ceph-mon[64248]: pgmap v137: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:35:53.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:53 smithi007 ceph-mon[61772]: pgmap v137: 161 pgs: 15 peering, 146 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:35:53.813 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:53 smithi149 podman[60028]: 2024-04-23 03:35:53.798824631 +0000 UTC m=+0.523541843 container remove 8ff96300f28cd102905468468fc67f73b942c92375a120bbbb31b9fa9155d345 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-23T03:35:54.172 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:53 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.5.service: Deactivated successfully. 2024-04-23T03:35:54.172 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:53 smithi149 systemd[1]: Stopped Ceph osd.5 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:35:54.172 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:53 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.5.service: Consumed 8.100s CPU time. 2024-04-23T03:35:54.172 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 systemd[1]: Starting Ceph osd.5 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:35:54.494 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-mon[54848]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-04-23T03:35:54.494 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 22/726 objects degraded (3.030%), 6 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:54.495 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 podman[60128]: 2024-04-23 03:35:54.140921999 +0000 UTC m=+0.017413214 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:35:54.495 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 podman[60128]: 2024-04-23 03:35:54.274845974 +0000 UTC m=+0.151337164 container create eced9f61494aa67417e424f02a93fd97af473409adc9a46d6304ced6b42b1067 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-23T03:35:54.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:54 smithi007 ceph-mon[64248]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-04-23T03:35:54.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:54 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 22/726 objects degraded (3.030%), 6 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:54.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:54 smithi007 ceph-mon[61772]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-04-23T03:35:54.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:54 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 22/726 objects degraded (3.030%), 6 pgs degraded (PG_DEGRADED) 2024-04-23T03:35:54.921 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 podman[60128]: 2024-04-23 03:35:54.494280964 +0000 UTC m=+0.370772147 container init eced9f61494aa67417e424f02a93fd97af473409adc9a46d6304ced6b42b1067 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-23T03:35:54.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 podman[60128]: 2024-04-23 03:35:54.49767972 +0000 UTC m=+0.374170912 container start eced9f61494aa67417e424f02a93fd97af473409adc9a46d6304ced6b42b1067 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.vendor=CentOS) 2024-04-23T03:35:54.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 podman[60128]: 2024-04-23 03:35:54.56079074 +0000 UTC m=+0.437281941 container attach eced9f61494aa67417e424f02a93fd97af473409adc9a46d6304ced6b42b1067 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416) 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-23T03:35:55.278 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 bash[60128]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate[60143]: --> ceph-volume raw activate successful for osd ID: 5 2024-04-23T03:35:55.279 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:54 smithi149 podman[60128]: 2024-04-23 03:35:54.998885653 +0000 UTC m=+0.875376855 container died eced9f61494aa67417e424f02a93fd97af473409adc9a46d6304ced6b42b1067 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df) 2024-04-23T03:35:55.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:55 smithi149 ceph-mon[54848]: pgmap v139: 161 pgs: 9 stale+active+clean, 6 active+undersized+degraded, 40 peering, 4 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/726 objects degraded (3.030%) 2024-04-23T03:35:55.672 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:55 smithi149 podman[60282]: 2024-04-23 03:35:55.464260374 +0000 UTC m=+0.460372444 container remove eced9f61494aa67417e424f02a93fd97af473409adc9a46d6304ced6b42b1067 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-activate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-56e81df) 2024-04-23T03:35:55.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:55 smithi007 ceph-mon[64248]: pgmap v139: 161 pgs: 9 stale+active+clean, 6 active+undersized+degraded, 40 peering, 4 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/726 objects degraded (3.030%) 2024-04-23T03:35:55.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:55 smithi007 ceph-mon[61772]: pgmap v139: 161 pgs: 9 stale+active+clean, 6 active+undersized+degraded, 40 peering, 4 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/726 objects degraded (3.030%) 2024-04-23T03:35:55.940 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:55 smithi149 podman[60348]: 2024-04-23 03:35:55.581443009 +0000 UTC m=+0.017931967 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:35:55.940 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:55 smithi149 podman[60348]: 2024-04-23 03:35:55.698383333 +0000 UTC m=+0.134872318 container create f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-23T03:35:55.941 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:55 smithi149 podman[60348]: 2024-04-23 03:35:55.917248492 +0000 UTC m=+0.353737434 container init f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T03:35:55.941 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:55 smithi149 podman[60348]: 2024-04-23 03:35:55.920613068 +0000 UTC m=+0.357102010 container start f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T03:35:55.941 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:55 smithi149 ceph-osd[60367]: -- 172.21.15.149:0/1342993011 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d8dcba5680 con 0x55d8dcb6f000 2024-04-23T03:35:56.421 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:56 smithi149 bash[60348]: f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 2024-04-23T03:35:56.422 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:56 smithi149 systemd[1]: Started Ceph osd.5 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:35:56.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:35:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:35:56] "GET /metrics HTTP/1.1" 200 38440 "" "Prometheus/2.33.4" 2024-04-23T03:35:56.847 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:35:56.463+0000 7fddab4227c0 -1 Falling back to public interface 2024-04-23T03:35:57.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:57 smithi149 ceph-mon[54848]: pgmap v140: 161 pgs: 9 stale+active+clean, 6 active+undersized+degraded, 40 peering, 4 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/726 objects degraded (3.030%) 2024-04-23T03:35:57.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:57 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:57.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:57 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:57.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:57 smithi007 ceph-mon[64248]: pgmap v140: 161 pgs: 9 stale+active+clean, 6 active+undersized+degraded, 40 peering, 4 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/726 objects degraded (3.030%) 2024-04-23T03:35:57.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:57 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:57.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:57 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:57.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:57 smithi007 ceph-mon[61772]: pgmap v140: 161 pgs: 9 stale+active+clean, 6 active+undersized+degraded, 40 peering, 4 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 22/726 objects degraded (3.030%) 2024-04-23T03:35:57.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:57 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:57.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:57 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:35:58.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:58 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:58.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:58 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:58.370 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:58 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:35:59.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:59 smithi149 ceph-mon[54848]: pgmap v141: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 81/726 objects degraded (11.157%) 2024-04-23T03:35:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:35:59 smithi149 ceph-mon[54848]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 7 pgs peering) 2024-04-23T03:35:59.422 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:35:59.202+0000 7fddab4227c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-04-23T03:35:59.422 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:35:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:35:59.230+0000 7fddab4227c0 -1 osd.5 130 log_to_monitors true 2024-04-23T03:35:59.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:59 smithi007 ceph-mon[61772]: pgmap v141: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 81/726 objects degraded (11.157%) 2024-04-23T03:35:59.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:35:59 smithi007 ceph-mon[61772]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 7 pgs peering) 2024-04-23T03:35:59.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:59 smithi007 ceph-mon[64248]: pgmap v141: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 81/726 objects degraded (11.157%) 2024-04-23T03:35:59.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:35:59 smithi007 ceph-mon[64248]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 7 pgs inactive, 7 pgs peering) 2024-04-23T03:36:00.421 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:36:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:36:00.125+0000 7fdd9c87a700 -1 osd.5 130 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:36:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:00 smithi149 ceph-mon[54848]: from='osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:36:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:00 smithi149 ceph-mon[54848]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:36:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:00 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:00.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:00 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:00.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[64248]: from='osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:36:00.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[64248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:36:00.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:00.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:00.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[61772]: from='osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:36:00.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[61772]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-23T03:36:00.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:00.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:00 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: pgmap v142: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 81/726 objects degraded (11.157%) 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: osdmap e133: 8 total, 7 up, 8 in 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 81/726 objects degraded (11.157%), 21 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:01 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: pgmap v142: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 81/726 objects degraded (11.157%) 2024-04-23T03:36:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T03:36:01.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: osdmap e133: 8 total, 7 up, 8 in 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 81/726 objects degraded (11.157%), 21 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: pgmap v142: 161 pgs: 21 active+undersized+degraded, 38 active+undersized, 102 active+clean; 585 KiB data, 190 MiB used, 715 GiB / 715 GiB avail; 81/726 objects degraded (11.157%) 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: osdmap e133: 8 total, 7 up, 8 in 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:01.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 81/726 objects degraded (11.157%), 21 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:01.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:01 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:02.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:02 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:36:02.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:02 smithi149 ceph-mon[54848]: osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447] boot 2024-04-23T03:36:02.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:02 smithi149 ceph-mon[54848]: osdmap e134: 8 total, 8 up, 8 in 2024-04-23T03:36:02.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:02 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:36:02.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:02 smithi149 ceph-mon[54848]: osdmap e135: 8 total, 8 up, 8 in 2024-04-23T03:36:02.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:36:02.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[64248]: osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447] boot 2024-04-23T03:36:02.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[64248]: osdmap e134: 8 total, 8 up, 8 in 2024-04-23T03:36:02.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:36:02.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[64248]: osdmap e135: 8 total, 8 up, 8 in 2024-04-23T03:36:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:36:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[61772]: osd.5 [v2:172.21.15.149:6808/2161761447,v1:172.21.15.149:6809/2161761447] boot 2024-04-23T03:36:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[61772]: osdmap e134: 8 total, 8 up, 8 in 2024-04-23T03:36:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-23T03:36:02.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:02 smithi007 ceph-mon[61772]: osdmap e135: 8 total, 8 up, 8 in 2024-04-23T03:36:03.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:03 smithi149 ceph-mon[54848]: pgmap v145: 161 pgs: 15 peering, 14 active+undersized+degraded, 30 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 65/726 objects degraded (8.953%) 2024-04-23T03:36:03.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:03 smithi007 ceph-mon[64248]: pgmap v145: 161 pgs: 15 peering, 14 active+undersized+degraded, 30 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 65/726 objects degraded (8.953%) 2024-04-23T03:36:03.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:03 smithi007 ceph-mon[61772]: pgmap v145: 161 pgs: 15 peering, 14 active+undersized+degraded, 30 active+undersized, 102 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 65/726 objects degraded (8.953%) 2024-04-23T03:36:04.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:04 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:36:04.531+0000 7f1c772de700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-23T03:36:05.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: pgmap v147: 161 pgs: 33 peering, 7 active+undersized+degraded, 14 active+undersized, 107 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 35/726 objects degraded (4.821%) 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:05 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: pgmap v147: 161 pgs: 33 peering, 7 active+undersized+degraded, 14 active+undersized, 107 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 35/726 objects degraded (4.821%) 2024-04-23T03:36:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: pgmap v147: 161 pgs: 33 peering, 7 active+undersized+degraded, 14 active+undersized, 107 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 35/726 objects degraded (4.821%) 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:05.584 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:05 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:06.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:06 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:06.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:06 smithi149 ceph-mon[54848]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-23T03:36:06.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:06 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 35/726 objects degraded (4.821%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:06.485 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:36:06] "GET /metrics HTTP/1.1" 200 38299 "" "Prometheus/2.33.4" 2024-04-23T03:36:06.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:06.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-mon[64248]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-23T03:36:06.485 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 35/726 objects degraded (4.821%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:06.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:06.485 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-mon[61772]: Upgrade: unsafe to stop osd(s) at this time (16 PGs are or would become offline) 2024-04-23T03:36:06.486 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:06 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 35/726 objects degraded (4.821%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:07 smithi149 ceph-mon[54848]: pgmap v148: 161 pgs: 33 peering, 128 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-23T03:36:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:07 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/726 objects degraded (4.821%), 7 pgs degraded) 2024-04-23T03:36:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:07 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:36:07.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:07 smithi007 ceph-mon[61772]: pgmap v148: 161 pgs: 33 peering, 128 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-23T03:36:07.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:07 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/726 objects degraded (4.821%), 7 pgs degraded) 2024-04-23T03:36:07.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:07 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:36:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:07 smithi007 ceph-mon[64248]: pgmap v148: 161 pgs: 33 peering, 128 active+clean; 585 KiB data, 205 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-04-23T03:36:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:07 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 35/726 objects degraded (4.821%), 7 pgs degraded) 2024-04-23T03:36:07.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:07 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:36:08.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:08 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:08.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:08 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:08 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:09.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:09 smithi149 ceph-mon[54848]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 413 B/s rd, 0 op/s 2024-04-23T03:36:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:09 smithi007 ceph-mon[64248]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 413 B/s rd, 0 op/s 2024-04-23T03:36:09.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:09 smithi007 ceph-mon[61772]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 413 B/s rd, 0 op/s 2024-04-23T03:36:11.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:11 smithi149 ceph-mon[54848]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-23T03:36:11.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:11 smithi007 ceph-mon[64248]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-23T03:36:11.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:11 smithi007 ceph-mon[61772]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 971 B/s rd, 0 op/s 2024-04-23T03:36:13.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:13 smithi149 ceph-mon[54848]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-23T03:36:13.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:13 smithi007 ceph-mon[64248]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-23T03:36:13.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:13 smithi007 ceph-mon[61772]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-04-23T03:36:15.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:15 smithi149 ceph-mon[54848]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:36:15.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:15 smithi007 ceph-mon[64248]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:36:15.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:15 smithi007 ceph-mon[61772]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:36:16.489 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:36:16.489 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:36:16] "GET /metrics HTTP/1.1" 200 38449 "" "Prometheus/2.33.4" 2024-04-23T03:36:16.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[64248]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:16.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:16.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:16.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:16.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[61772]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:16.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:16.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:16 smithi149 ceph-mon[54848]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:16.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:17.193 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (5m) 84s ago 10m 24.2M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (5m) 17s ago 9m 54.8M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (4m) 84s ago 9m 52.9M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (4m) 17s ago 14m 452M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (5m) 84s ago 16m 507M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (4m) 84s ago 16m 51.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (3m) 17s ago 14m 36.5M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (3m) 84s ago 15m 44.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 84s ago 10m 17.1M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (5m) 17s ago 10m 16.8M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (2m) 84s ago 14m 44.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (2m) 84s ago 13m 43.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:36:17.194 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (109s) 84s ago 13m 44.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (89s) 84s ago 13m 12.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (54s) 17s ago 12m 45.0M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (21s) 17s ago 12m 12.7M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 f88b3de0b1f7 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (12m) 17s ago 12m 59.0M 1454M 17.2.0 e1d6a67b021e 92ccebe9292e 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (11m) 17s ago 11m 63.0M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (4m) 17s ago 11m 74.1M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (9m) 84s ago 9m 74.3M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (9m) 17s ago 9m 76.7M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (9m) 84s ago 9m 75.8M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:36:17.195 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (9m) 17s ago 9m 75.8M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:36:17.609 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:36:17.609 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:36:17.609 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:36:17.609 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:17.609 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 6 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 11 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:36:17.610 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:36:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[61772]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[61772]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[61772]: from='client.44436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[64248]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[64248]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:17 smithi007 ceph-mon[64248]: from='client.44436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:17 smithi149 ceph-mon[54848]: from='client.54355 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:17 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:17 smithi149 ceph-mon[54848]: from='client.44430 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:17 smithi149 ceph-mon[54848]: from='client.44436 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:36:17.966 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:36:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:36:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "11/23 daemons upgraded", 2024-04-23T03:36:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:36:17.967 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:36:17.967 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:36:18.387 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:36:18.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[64248]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:18.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/4001295181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:18.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[64248]: from='client.44448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:18.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3127431100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:36:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[61772]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/4001295181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[61772]: from='client.44448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:18 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3127431100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:36:18.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:18 smithi149 ceph-mon[54848]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:18 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/4001295181' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:18 smithi149 ceph-mon[54848]: from='client.44448 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:18 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3127431100' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:36:19.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:19 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:19.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:19 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:19.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:19 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:20 smithi149 ceph-mon[54848]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:36:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:20 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:20.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:20 smithi149 ceph-mon[54848]: Upgrade: osd.6 is safe to restart 2024-04-23T03:36:20.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:20 smithi007 ceph-mon[61772]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:36:20.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:20 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:20.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:20 smithi007 ceph-mon[61772]: Upgrade: osd.6 is safe to restart 2024-04-23T03:36:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:20 smithi007 ceph-mon[64248]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:36:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:20 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-04-23T03:36:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:20 smithi007 ceph-mon[64248]: Upgrade: osd.6 is safe to restart 2024-04-23T03:36:22.235 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:22 smithi149 ceph-mon[54848]: Upgrade: Updating osd.6 2024-04-23T03:36:22.235 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:22.235 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:36:22.235 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:22.235 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:22 smithi149 ceph-mon[54848]: Deploying daemon osd.6 on smithi149 2024-04-23T03:36:22.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[61772]: Upgrade: Updating osd.6 2024-04-23T03:36:22.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[61772]: Deploying daemon osd.6 on smithi149 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[64248]: Upgrade: Updating osd.6 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:22.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:22 smithi007 ceph-mon[64248]: Deploying daemon osd.6 on smithi149 2024-04-23T03:36:23.349 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:23 smithi149 ceph-mon[54848]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:23.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:23 smithi007 ceph-mon[61772]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:23.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:23 smithi007 ceph-mon[64248]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:36:24.272 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:23 smithi149 systemd[1]: Stopping Ceph osd.6 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:36:24.672 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:24 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[31947]: 2024-04-23T03:36:24.271+0000 7f0d1b6c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:36:24.672 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:24 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[31947]: 2024-04-23T03:36:24.271+0000 7f0d1b6c5700 -1 osd.6 135 *** Got signal Terminated *** 2024-04-23T03:36:24.672 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:24 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[31947]: 2024-04-23T03:36:24.271+0000 7f0d1b6c5700 -1 osd.6 135 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:36:25.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:25 smithi149 ceph-mon[54848]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:36:25.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:25 smithi149 ceph-mon[54848]: osd.6 marked itself down and dead 2024-04-23T03:36:25.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:25 smithi007 ceph-mon[64248]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:36:25.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:25 smithi007 ceph-mon[64248]: osd.6 marked itself down and dead 2024-04-23T03:36:25.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:25 smithi007 ceph-mon[61772]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:36:25.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:25 smithi007 ceph-mon[61772]: osd.6 marked itself down and dead 2024-04-23T03:36:25.814 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:25 smithi149 podman[62242]: 2024-04-23 03:36:25.558228335 +0000 UTC m=+1.404548879 container died 92ccebe9292eb05e3fcf4dad40a4fcf8d630df99136a66eb30235ef29047fbe1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , release=754, architecture=x86_64, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, ceph=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0) 2024-04-23T03:36:26.121 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:25 smithi149 podman[62242]: 2024-04-23 03:36:25.912859486 +0000 UTC m=+1.759180037 container cleanup 92ccebe9292eb05e3fcf4dad40a4fcf8d630df99136a66eb30235ef29047fbe1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-23T03:36:26.121 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:25 smithi149 bash[62242]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6 2024-04-23T03:36:26.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:26 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:36:26.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:26 smithi149 ceph-mon[54848]: osdmap e136: 8 total, 7 up, 8 in 2024-04-23T03:36:26.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:26 smithi149 podman[62260]: 2024-04-23 03:36:26.222489059 +0000 UTC m=+0.659320635 container remove 92ccebe9292eb05e3fcf4dad40a4fcf8d630df99136a66eb30235ef29047fbe1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-23T03:36:26.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:36:26] "GET /metrics HTTP/1.1" 200 38449 "" "Prometheus/2.33.4" 2024-04-23T03:36:26.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:26 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:36:26.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:26 smithi007 ceph-mon[61772]: osdmap e136: 8 total, 7 up, 8 in 2024-04-23T03:36:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:26 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:36:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:26 smithi007 ceph-mon[64248]: osdmap e136: 8 total, 7 up, 8 in 2024-04-23T03:36:26.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:26 smithi149 podman[62353]: 2024-04-23 03:36:26.484163881 +0000 UTC m=+0.018660673 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:36:26.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:26 smithi149 podman[62353]: 2024-04-23 03:36:26.58183447 +0000 UTC m=+0.116331247 container create 2cb29ccd4a971a28dbac18f0b8c2ea837be246ee0d7b491ee3f8fd66f611e89b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-23T03:36:26.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:26 smithi149 podman[62353]: 2024-04-23 03:36:26.767646224 +0000 UTC m=+0.302143001 container init 2cb29ccd4a971a28dbac18f0b8c2ea837be246ee0d7b491ee3f8fd66f611e89b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:36:26.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:26 smithi149 podman[62353]: 2024-04-23 03:36:26.770890642 +0000 UTC m=+0.305387428 container start 2cb29ccd4a971a28dbac18f0b8c2ea837be246ee0d7b491ee3f8fd66f611e89b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-23T03:36:26.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:26 smithi149 podman[62353]: 2024-04-23 03:36:26.825699455 +0000 UTC m=+0.360196247 container attach 2cb29ccd4a971a28dbac18f0b8c2ea837be246ee0d7b491ee3f8fd66f611e89b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-23T03:36:27.337 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:27 smithi149 podman[62353]: 2024-04-23 03:36:27.069866706 +0000 UTC m=+0.604363485 container died 2cb29ccd4a971a28dbac18f0b8c2ea837be246ee0d7b491ee3f8fd66f611e89b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T03:36:27.547 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:27 smithi149 ceph-mon[54848]: pgmap v159: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:36:27.547 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:27 smithi149 ceph-mon[54848]: osdmap e137: 8 total, 7 up, 8 in 2024-04-23T03:36:27.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:27 smithi007 ceph-mon[64248]: pgmap v159: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:36:27.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:27 smithi007 ceph-mon[64248]: osdmap e137: 8 total, 7 up, 8 in 2024-04-23T03:36:27.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:27 smithi007 ceph-mon[61772]: pgmap v159: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:36:27.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:27 smithi007 ceph-mon[61772]: osdmap e137: 8 total, 7 up, 8 in 2024-04-23T03:36:28.136 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:27 smithi149 podman[62379]: 2024-04-23 03:36:27.687851819 +0000 UTC m=+0.612911424 container remove 2cb29ccd4a971a28dbac18f0b8c2ea837be246ee0d7b491ee3f8fd66f611e89b (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T03:36:28.137 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:27 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.6.service: Deactivated successfully. 2024-04-23T03:36:28.137 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:27 smithi149 systemd[1]: Stopped Ceph osd.6 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:36:28.137 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:27 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.6.service: Consumed 7.819s CPU time. 2024-04-23T03:36:28.137 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:27 smithi149 systemd[1]: Starting Ceph osd.6 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:36:28.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:28.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 podman[62484]: 2024-04-23 03:36:28.040534121 +0000 UTC m=+0.018669575 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:36:28.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 podman[62484]: 2024-04-23 03:36:28.171952819 +0000 UTC m=+0.150088300 container create c473a924f813427ca73a5ac6adfab0290bfecddfb1e5483bb32dc926e83f8d67 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-56e81df, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T03:36:28.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 podman[62484]: 2024-04-23 03:36:28.374516752 +0000 UTC m=+0.352652190 container init c473a924f813427ca73a5ac6adfab0290bfecddfb1e5483bb32dc926e83f8d67 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:36:28.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 podman[62484]: 2024-04-23 03:36:28.378677975 +0000 UTC m=+0.356813416 container start c473a924f813427ca73a5ac6adfab0290bfecddfb1e5483bb32dc926e83f8d67 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-04-23T03:36:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:28 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:28 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:28.856 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 podman[62484]: 2024-04-23 03:36:28.432753905 +0000 UTC m=+0.410889363 container attach c473a924f813427ca73a5ac6adfab0290bfecddfb1e5483bb32dc926e83f8d67 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-23T03:36:29.138 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 bash[62484]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate[62499]: --> ceph-volume raw activate successful for osd ID: 6 2024-04-23T03:36:29.139 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:28 smithi149 podman[62484]: 2024-04-23 03:36:28.881037861 +0000 UTC m=+0.859173303 container died c473a924f813427ca73a5ac6adfab0290bfecddfb1e5483bb32dc926e83f8d67 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T03:36:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:29 smithi149 ceph-mon[54848]: pgmap v161: 161 pgs: 9 active+undersized+degraded, 3 stale+active+clean, 24 active+undersized, 125 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 56/726 objects degraded (7.713%) 2024-04-23T03:36:29.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:29 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 56/726 objects degraded (7.713%), 9 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:29.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:29 smithi007 ceph-mon[61772]: pgmap v161: 161 pgs: 9 active+undersized+degraded, 3 stale+active+clean, 24 active+undersized, 125 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 56/726 objects degraded (7.713%) 2024-04-23T03:36:29.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:29 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 56/726 objects degraded (7.713%), 9 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:29.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:29 smithi007 ceph-mon[64248]: pgmap v161: 161 pgs: 9 active+undersized+degraded, 3 stale+active+clean, 24 active+undersized, 125 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 56/726 objects degraded (7.713%) 2024-04-23T03:36:29.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:29 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 56/726 objects degraded (7.713%), 9 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:29.745 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:29 smithi149 podman[62627]: 2024-04-23 03:36:29.486510986 +0000 UTC m=+0.596560081 container remove c473a924f813427ca73a5ac6adfab0290bfecddfb1e5483bb32dc926e83f8d67 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-activate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-56e81df, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:36:29.745 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:29 smithi149 podman[62689]: 2024-04-23 03:36:29.607776031 +0000 UTC m=+0.018582301 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:36:30.065 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:29 smithi149 podman[62689]: 2024-04-23 03:36:29.745740966 +0000 UTC m=+0.156547221 container create ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, io.buildah.version=1.33.5) 2024-04-23T03:36:30.065 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:29 smithi149 podman[62689]: 2024-04-23 03:36:29.964646969 +0000 UTC m=+0.375453223 container init ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True) 2024-04-23T03:36:30.065 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:29 smithi149 podman[62689]: 2024-04-23 03:36:29.96762589 +0000 UTC m=+0.378432145 container start ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-23T03:36:30.065 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:29 smithi149 ceph-osd[62708]: -- 172.21.15.149:0/2169559352 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d58ddc3680 con 0x55d58dd8cc00 2024-04-23T03:36:30.421 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:30 smithi149 bash[62689]: ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e 2024-04-23T03:36:30.421 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:30 smithi149 systemd[1]: Started Ceph osd.6 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:36:30.878 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:30 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:36:30.510+0000 7f51f95c67c0 -1 Falling back to public interface 2024-04-23T03:36:31.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:31 smithi149 ceph-mon[54848]: pgmap v162: 161 pgs: 15 active+undersized+degraded, 2 stale+active+clean, 29 active+undersized, 115 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 74/726 objects degraded (10.193%) 2024-04-23T03:36:31.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:31 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:31 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:31 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:31 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[64248]: pgmap v162: 161 pgs: 15 active+undersized+degraded, 2 stale+active+clean, 29 active+undersized, 115 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 74/726 objects degraded (10.193%) 2024-04-23T03:36:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[61772]: pgmap v162: 161 pgs: 15 active+undersized+degraded, 2 stale+active+clean, 29 active+undersized, 115 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 74/726 objects degraded (10.193%) 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:31.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:31 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:33.369 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:33 smithi149 ceph-mon[54848]: pgmap v163: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:33.369 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:36:33.215+0000 7f51f95c67c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-04-23T03:36:33.369 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:33 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:36:33.243+0000 7f51f95c67c0 -1 osd.6 135 log_to_monitors true 2024-04-23T03:36:33.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:33 smithi007 ceph-mon[64248]: pgmap v163: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:33.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:33 smithi007 ceph-mon[61772]: pgmap v163: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:34.386 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:34 smithi149 ceph-mon[54848]: from='osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:36:34.386 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:34 smithi149 ceph-mon[54848]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:36:34.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:34 smithi007 ceph-mon[64248]: from='osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:36:34.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:34 smithi007 ceph-mon[64248]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:36:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:34 smithi007 ceph-mon[61772]: from='osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:36:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:34 smithi007 ceph-mon[61772]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-23T03:36:34.671 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:36:34 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:36:34.603+0000 7f51e6a16700 -1 osd.6 135 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: pgmap v164: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: osdmap e138: 8 total, 7 up, 8 in 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: from='osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:35.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:35 smithi149 ceph-mon[54848]: from='osd.6 ' entity='osd.6' 2024-04-23T03:36:35.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: pgmap v164: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:35.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T03:36:35.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: osdmap e138: 8 total, 7 up, 8 in 2024-04-23T03:36:35.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: from='osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[64248]: from='osd.6 ' entity='osd.6' 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: pgmap v164: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 206 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: osdmap e138: 8 total, 7 up, 8 in 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: from='osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:35.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:35 smithi007 ceph-mon[61772]: from='osd.6 ' entity='osd.6' 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 86/726 objects degraded (11.846%), 18 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828] boot 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: osdmap e139: 8 total, 8 up, 8 in 2024-04-23T03:36:36.377 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:36 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:36:36.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:36:36] "GET /metrics HTTP/1.1" 200 38453 "" "Prometheus/2.33.4" 2024-04-23T03:36:36.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 86/726 objects degraded (11.846%), 18 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:36.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828] boot 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: osdmap e139: 8 total, 8 up, 8 in 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 86/726 objects degraded (11.846%), 18 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: osd.6 [v2:172.21.15.149:6816/1407837828,v1:172.21.15.149:6817/1407837828] boot 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: osdmap e139: 8 total, 8 up, 8 in 2024-04-23T03:36:36.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:36 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-23T03:36:37.149 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:37 smithi149 ceph-mon[54848]: pgmap v166: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:37.149 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:37 smithi149 ceph-mon[54848]: osdmap e140: 8 total, 8 up, 8 in 2024-04-23T03:36:37.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:37 smithi007 ceph-mon[64248]: pgmap v166: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:37.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:37 smithi007 ceph-mon[64248]: osdmap e140: 8 total, 8 up, 8 in 2024-04-23T03:36:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:37 smithi007 ceph-mon[61772]: pgmap v166: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 86/726 objects degraded (11.846%) 2024-04-23T03:36:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:37 smithi007 ceph-mon[61772]: osdmap e140: 8 total, 8 up, 8 in 2024-04-23T03:36:38.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:38 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:38.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:38 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:38.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:38 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:39.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:36:38.738+0000 7f1c772de700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: pgmap v169: 161 pgs: 18 peering, 11 active+undersized+degraded, 19 active+undersized, 113 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 42/726 objects degraded (5.785%) 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:39 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:39.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: pgmap v169: 161 pgs: 18 peering, 11 active+undersized+degraded, 19 active+undersized, 113 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 42/726 objects degraded (5.785%) 2024-04-23T03:36:39.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:39.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: pgmap v169: 161 pgs: 18 peering, 11 active+undersized+degraded, 19 active+undersized, 113 active+clean; 585 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 42/726 objects degraded (5.785%) 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:39.584 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:39 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:40.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:40 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:40.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:40 smithi149 ceph-mon[54848]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-23T03:36:40.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:40 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:40.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:40 smithi007 ceph-mon[64248]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-23T03:36:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:40 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:40.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:40 smithi007 ceph-mon[61772]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-04-23T03:36:41.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:41 smithi149 ceph-mon[54848]: pgmap v170: 161 pgs: 18 peering, 2 active+undersized+degraded, 5 active+undersized, 136 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 11/726 objects degraded (1.515%) 2024-04-23T03:36:41.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:41 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 11/726 objects degraded (1.515%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:41 smithi007 ceph-mon[64248]: pgmap v170: 161 pgs: 18 peering, 2 active+undersized+degraded, 5 active+undersized, 136 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 11/726 objects degraded (1.515%) 2024-04-23T03:36:41.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:41 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 11/726 objects degraded (1.515%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:41 smithi007 ceph-mon[61772]: pgmap v170: 161 pgs: 18 peering, 2 active+undersized+degraded, 5 active+undersized, 136 active+clean; 585 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 11/726 objects degraded (1.515%) 2024-04-23T03:36:41.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:41 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 11/726 objects degraded (1.515%), 2 pgs degraded (PG_DEGRADED) 2024-04-23T03:36:42.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:42 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/726 objects degraded (1.515%), 2 pgs degraded) 2024-04-23T03:36:42.476 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:42 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:36:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:42 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/726 objects degraded (1.515%), 2 pgs degraded) 2024-04-23T03:36:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:42 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:36:42.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:42 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 11/726 objects degraded (1.515%), 2 pgs degraded) 2024-04-23T03:36:42.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:42 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:36:43.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:43 smithi007 ceph-mon[61772]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s 2024-04-23T03:36:43.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:43 smithi007 ceph-mon[64248]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s 2024-04-23T03:36:43.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:43 smithi149 ceph-mon[54848]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 414 B/s rd, 0 op/s 2024-04-23T03:36:45.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:45 smithi007 ceph-mon[61772]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:36:45.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:45 smithi007 ceph-mon[64248]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:36:45.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:45 smithi149 ceph-mon[54848]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:36:46.539 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:36:46] "GET /metrics HTTP/1.1" 200 38457 "" "Prometheus/2.33.4" 2024-04-23T03:36:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-mon[61772]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-04-23T03:36:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:46.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:46.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-mon[64248]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-04-23T03:36:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:46.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:46.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:46 smithi149 ceph-mon[54848]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 0 op/s 2024-04-23T03:36:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:46.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:36:47.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:47 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:47.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:47 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:47.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:47 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:48.770 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:36:48.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:48 smithi007 ceph-mon[64248]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 749 B/s rd, 0 op/s 2024-04-23T03:36:48.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:48 smithi007 ceph-mon[61772]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 749 B/s rd, 0 op/s 2024-04-23T03:36:48.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:48 smithi149 ceph-mon[54848]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 749 B/s rd, 0 op/s 2024-04-23T03:36:49.499 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:36:49.499 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (5m) 116s ago 10m 24.2M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:36:49.499 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (5m) 15s ago 10m 55.1M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:36:49.499 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (5m) 116s ago 9m 52.9M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (4m) 15s ago 15m 453M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (5m) 116s ago 16m 507M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (4m) 116s ago 17m 51.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (4m) 15s ago 15m 38.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (4m) 116s ago 15m 44.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (5m) 116s ago 11m 17.1M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (5m) 15s ago 11m 19.5M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (3m) 116s ago 14m 44.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (2m) 116s ago 14m 43.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (2m) 116s ago 14m 44.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (2m) 116s ago 13m 12.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (87s) 15s ago 13m 45.4M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (53s) 15s ago 12m 43.7M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 f88b3de0b1f7 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (19s) 15s ago 12m 14.8M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 ec30c8d5356b 2024-04-23T03:36:49.500 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (12m) 15s ago 12m 63.4M 1454M 17.2.0 e1d6a67b021e cb06a815489c 2024-04-23T03:36:49.501 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (5m) 15s ago 11m 76.2M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:36:49.501 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (10m) 116s ago 10m 74.3M - 17.2.0 e1d6a67b021e 80022916109e 2024-04-23T03:36:49.501 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (9m) 15s ago 9m 76.8M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:36:49.501 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (9m) 116s ago 9m 75.8M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:36:49.501 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (9m) 15s ago 9m 76.0M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:36:49.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:49 smithi007 ceph-mon[64248]: from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:49.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:49 smithi007 ceph-mon[64248]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:49.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:49 smithi007 ceph-mon[61772]: from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:49.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:49 smithi007 ceph-mon[61772]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:49.919 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:36:49.919 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:36:49.919 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:36:49.919 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 7 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 12 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:36:49.920 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:36:49.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:49 smithi149 ceph-mon[54848]: from='client.54397 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:49.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:49 smithi149 ceph-mon[54848]: from='client.54403 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:50.268 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr" 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "12/23 daemons upgraded", 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading osd daemons", 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:36:50.269 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:36:50.683 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:36:50.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[61772]: from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:50.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[61772]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-23T03:36:50.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3372357904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:50.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[61772]: from='client.44490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:50.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[64248]: from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:50.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[64248]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-23T03:36:50.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3372357904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:50.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:50 smithi007 ceph-mon[64248]: from='client.44490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:50.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:50 smithi149 ceph-mon[54848]: from='client.54409 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:50.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:50 smithi149 ceph-mon[54848]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s 2024-04-23T03:36:50.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:50 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3372357904' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:36:50.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:50 smithi149 ceph-mon[54848]: from='client.44490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:36:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:51 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/4023260940' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:36:51.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:51 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:51.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:51 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/4023260940' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:36:51.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:51 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:51.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:51 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/4023260940' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:36:51.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:51 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:53.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:52 smithi149 ceph-mon[54848]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-04-23T03:36:53.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:52 smithi007 ceph-mon[64248]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-04-23T03:36:53.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:52 smithi007 ceph-mon[61772]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 36 op/s 2024-04-23T03:36:54.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:53 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:54.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:53 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:54.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:53 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:55.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:54 smithi149 ceph-mon[54848]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-23T03:36:55.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:54 smithi149 ceph-mon[54848]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:55.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:54 smithi149 ceph-mon[54848]: Upgrade: osd.7 is safe to restart 2024-04-23T03:36:55.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:54 smithi007 ceph-mon[64248]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-23T03:36:55.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:54 smithi007 ceph-mon[64248]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:55.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:54 smithi007 ceph-mon[64248]: Upgrade: osd.7 is safe to restart 2024-04-23T03:36:55.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:54 smithi007 ceph-mon[61772]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 92 op/s 2024-04-23T03:36:55.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:54 smithi007 ceph-mon[61772]: from='mon.1 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-04-23T03:36:55.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:54 smithi007 ceph-mon[61772]: Upgrade: osd.7 is safe to restart 2024-04-23T03:36:56.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:36:56] "GET /metrics HTTP/1.1" 200 38457 "" "Prometheus/2.33.4" 2024-04-23T03:36:56.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[64248]: Upgrade: Updating osd.7 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[64248]: Deploying daemon osd.7 on smithi149 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[61772]: Upgrade: Updating osd.7 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:56.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:56 smithi007 ceph-mon[61772]: Deploying daemon osd.7 on smithi149 2024-04-23T03:36:56.666 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:56 smithi149 ceph-mon[54848]: Upgrade: Updating osd.7 2024-04-23T03:36:56.667 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:56 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:36:56.667 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:56 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-23T03:36:56.667 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:56 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:36:56.667 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:56 smithi149 ceph-mon[54848]: Deploying daemon osd.7 on smithi149 2024-04-23T03:36:57.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:57 smithi007 ceph-mon[61772]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-04-23T03:36:57.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:57 smithi007 ceph-mon[64248]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-04-23T03:36:57.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:57 smithi149 ceph-mon[54848]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 0 B/s wr, 108 op/s 2024-04-23T03:36:58.554 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:58 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:58.555 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:36:58 smithi149 systemd[1]: Stopping Ceph osd.7 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:36:58.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:58 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:58.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:58 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:36:58.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:36:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[34862]: 2024-04-23T03:36:58.552+0000 7f96b1382700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:36:58.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:36:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[34862]: 2024-04-23T03:36:58.552+0000 7f96b1382700 -1 osd.7 140 *** Got signal Terminated *** 2024-04-23T03:36:58.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:36:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[34862]: 2024-04-23T03:36:58.552+0000 7f96b1382700 -1 osd.7 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:36:59.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:59 smithi149 ceph-mon[54848]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-23T03:36:59.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:36:59 smithi149 ceph-mon[54848]: osd.7 marked itself down and dead 2024-04-23T03:36:59.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:59 smithi007 ceph-mon[64248]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-23T03:36:59.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:36:59 smithi007 ceph-mon[64248]: osd.7 marked itself down and dead 2024-04-23T03:36:59.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:59 smithi007 ceph-mon[61772]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s 2024-04-23T03:36:59.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:36:59 smithi007 ceph-mon[61772]: osd.7 marked itself down and dead 2024-04-23T03:37:00.087 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:36:59 smithi149 podman[64584]: 2024-04-23 03:36:59.814754885 +0000 UTC m=+1.384316242 container died cb06a815489c2b95db1830bfed2dba3e0a43e276daa0101a8322e77ceab9cbdb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, name=centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD) 2024-04-23T03:37:00.339 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:00 smithi149 podman[64584]: 2024-04-23 03:37:00.244638183 +0000 UTC m=+1.814199585 container cleanup cb06a815489c2b95db1830bfed2dba3e0a43e276daa0101a8322e77ceab9cbdb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, ceph=True, name=centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, version=8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-04-23T03:37:00.339 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:00 smithi149 bash[64584]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7 2024-04-23T03:37:00.671 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:00 smithi149 podman[64612]: 2024-04-23 03:37:00.421189918 +0000 UTC m=+0.605739462 container remove cb06a815489c2b95db1830bfed2dba3e0a43e276daa0101a8322e77ceab9cbdb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, version=8, maintainer=Guillaume Abrioux , vcs-type=git, architecture=x86_64, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=754, name=centos-stream) 2024-04-23T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:00 smithi149 ceph-mon[54848]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:00 smithi149 ceph-mon[54848]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-23T03:37:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:00 smithi149 ceph-mon[54848]: osdmap e141: 8 total, 7 up, 8 in 2024-04-23T03:37:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:00 smithi007 ceph-mon[64248]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:37:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:00 smithi007 ceph-mon[64248]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-23T03:37:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:00 smithi007 ceph-mon[64248]: osdmap e141: 8 total, 7 up, 8 in 2024-04-23T03:37:00.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:00 smithi007 ceph-mon[61772]: Health check failed: 1 osds down (OSD_DOWN) 2024-04-23T03:37:00.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:00 smithi007 ceph-mon[61772]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-04-23T03:37:00.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:00 smithi007 ceph-mon[61772]: osdmap e141: 8 total, 7 up, 8 in 2024-04-23T03:37:01.141 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:00 smithi149 podman[64701]: 2024-04-23 03:37:00.863859479 +0000 UTC m=+0.114061338 container create a7e3fdf0562644d954c0090c01afc2260060157116b9599a71b813616ae86b0c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240416, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-23T03:37:01.141 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:00 smithi149 podman[64701]: 2024-04-23 03:37:00.768298339 +0000 UTC m=+0.018500165 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:37:01.141 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 podman[64701]: 2024-04-23 03:37:01.066386652 +0000 UTC m=+0.316588466 container init a7e3fdf0562644d954c0090c01afc2260060157116b9599a71b813616ae86b0c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:37:01.142 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 podman[64701]: 2024-04-23 03:37:01.069697432 +0000 UTC m=+0.319899248 container start a7e3fdf0562644d954c0090c01afc2260060157116b9599a71b813616ae86b0c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416) 2024-04-23T03:37:01.142 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 podman[64701]: 2024-04-23 03:37:01.141356386 +0000 UTC m=+0.391558218 container attach a7e3fdf0562644d954c0090c01afc2260060157116b9599a71b813616ae86b0c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-23T03:37:01.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:01 smithi149 ceph-mon[54848]: pgmap v181: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 141 op/s 2024-04-23T03:37:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:01 smithi149 ceph-mon[54848]: osdmap e142: 8 total, 7 up, 8 in 2024-04-23T03:37:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:01 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:01.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 podman[64701]: 2024-04-23 03:37:01.363635542 +0000 UTC m=+0.613837350 container died a7e3fdf0562644d954c0090c01afc2260060157116b9599a71b813616ae86b0c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, RELEASE=main-56e81df) 2024-04-23T03:37:01.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[64248]: pgmap v181: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 141 op/s 2024-04-23T03:37:01.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[64248]: osdmap e142: 8 total, 7 up, 8 in 2024-04-23T03:37:01.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:01.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:01.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[61772]: pgmap v181: 161 pgs: 23 stale+active+clean, 138 active+clean; 585 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s rd, 0 B/s wr, 141 op/s 2024-04-23T03:37:01.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[61772]: osdmap e142: 8 total, 7 up, 8 in 2024-04-23T03:37:01.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:01.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:01.906 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 podman[64727]: 2024-04-23 03:37:01.88628409 +0000 UTC m=+0.512986274 container remove a7e3fdf0562644d954c0090c01afc2260060157116b9599a71b813616ae86b0c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T03:37:02.172 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.7.service: Deactivated successfully. 2024-04-23T03:37:02.172 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 systemd[1]: Stopped Ceph osd.7 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:37:02.172 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:01 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.7.service: Consumed 9.578s CPU time. 2024-04-23T03:37:02.172 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:02 smithi149 systemd[1]: Starting Ceph osd.7 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:37:02.591 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:02 smithi149 ceph-mon[54848]: pgmap v183: 161 pgs: 7 active+undersized+degraded, 10 peering, 13 stale+active+clean, 3 active+undersized, 128 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 61 op/s; 39/726 objects degraded (5.372%) 2024-04-23T03:37:02.591 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:02 smithi149 ceph-mon[54848]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-23T03:37:02.591 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:02 smithi149 ceph-mon[54848]: Health check failed: Degraded data redundancy: 39/726 objects degraded (5.372%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:37:02.591 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:02 smithi149 podman[64830]: 2024-04-23 03:37:02.320777882 +0000 UTC m=+0.107362099 container create 328556f29f1b9083168617d58beace722e9fdcf5f499e242ad0259e79e6151e3 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.schema-version=1.0) 2024-04-23T03:37:02.591 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:02 smithi149 podman[64830]: 2024-04-23 03:37:02.231906531 +0000 UTC m=+0.018490772 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:37:02.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:02 smithi007 ceph-mon[64248]: pgmap v183: 161 pgs: 7 active+undersized+degraded, 10 peering, 13 stale+active+clean, 3 active+undersized, 128 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 61 op/s; 39/726 objects degraded (5.372%) 2024-04-23T03:37:02.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:02 smithi007 ceph-mon[64248]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-23T03:37:02.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:02 smithi007 ceph-mon[64248]: Health check failed: Degraded data redundancy: 39/726 objects degraded (5.372%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:37:02.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:02 smithi007 ceph-mon[61772]: pgmap v183: 161 pgs: 7 active+undersized+degraded, 10 peering, 13 stale+active+clean, 3 active+undersized, 128 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 61 op/s; 39/726 objects degraded (5.372%) 2024-04-23T03:37:02.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:02 smithi007 ceph-mon[61772]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-04-23T03:37:02.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:02 smithi007 ceph-mon[61772]: Health check failed: Degraded data redundancy: 39/726 objects degraded (5.372%), 7 pgs degraded (PG_DEGRADED) 2024-04-23T03:37:02.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:02 smithi149 podman[64830]: 2024-04-23 03:37:02.591175958 +0000 UTC m=+0.377760179 container init 328556f29f1b9083168617d58beace722e9fdcf5f499e242ad0259e79e6151e3 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True) 2024-04-23T03:37:02.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:02 smithi149 podman[64830]: 2024-04-23 03:37:02.594917456 +0000 UTC m=+0.381501681 container start 328556f29f1b9083168617d58beace722e9fdcf5f499e242ad0259e79e6151e3 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:37:02.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:02 smithi149 podman[64830]: 2024-04-23 03:37:02.698734624 +0000 UTC m=+0.485318851 container attach 328556f29f1b9083168617d58beace722e9fdcf5f499e242ad0259e79e6151e3 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-23T03:37:03.325 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 bash[64830]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate[64845]: --> ceph-volume raw activate successful for osd ID: 7 2024-04-23T03:37:03.326 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 podman[64830]: 2024-04-23 03:37:03.096260446 +0000 UTC m=+0.882844655 container died 328556f29f1b9083168617d58beace722e9fdcf5f499e242ad0259e79e6151e3 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=main-56e81df, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-23T03:37:03.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 podman[64984]: 2024-04-23 03:37:03.643723924 +0000 UTC m=+0.542426300 container remove 328556f29f1b9083168617d58beace722e9fdcf5f499e242ad0259e79e6151e3 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-04-23T03:37:03.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 podman[65049]: 2024-04-23 03:37:03.852960504 +0000 UTC m=+0.111218645 container create 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, CEPH_POINT_RELEASE=) 2024-04-23T03:37:03.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:03 smithi149 podman[65049]: 2024-04-23 03:37:03.760068059 +0000 UTC m=+0.018326146 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:37:04.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:04 smithi149 podman[65049]: 2024-04-23 03:37:04.097095649 +0000 UTC m=+0.355353724 container init 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD) 2024-04-23T03:37:04.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:04 smithi149 podman[65049]: 2024-04-23 03:37:04.100182104 +0000 UTC m=+0.358440180 container start 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:37:04.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:04 smithi149 ceph-osd[65068]: -- 172.21.15.149:0/1270537035 <== mon.2 v2:172.21.15.149:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556c06be3680 con 0x556c06bad400 2024-04-23T03:37:04.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:04 smithi149 bash[65049]: 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 2024-04-23T03:37:04.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:04 smithi149 systemd[1]: Started Ceph osd.7 for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:37:04.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:37:04.647+0000 7f3e3f9e47c0 -1 Falling back to public interface 2024-04-23T03:37:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:05 smithi149 ceph-mon[54848]: pgmap v184: 161 pgs: 22 active+undersized+degraded, 10 peering, 5 stale+active+clean, 26 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 95 op/s; 78/726 objects degraded (10.744%) 2024-04-23T03:37:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:05.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:05 smithi007 ceph-mon[64248]: pgmap v184: 161 pgs: 22 active+undersized+degraded, 10 peering, 5 stale+active+clean, 26 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 95 op/s; 78/726 objects degraded (10.744%) 2024-04-23T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:05.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:05 smithi007 ceph-mon[61772]: pgmap v184: 161 pgs: 22 active+undersized+degraded, 10 peering, 5 stale+active+clean, 26 active+undersized, 98 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 95 op/s; 78/726 objects degraded (10.744%) 2024-04-23T03:37:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:05.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:06.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:06 smithi007 ceph-mon[64248]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-23T03:37:06.484 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:06 smithi007 ceph-mon[61772]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-23T03:37:06.484 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:37:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:37:06] "GET /metrics HTTP/1.1" 200 38535 "" "Prometheus/2.33.4" 2024-04-23T03:37:06.546 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:06 smithi149 ceph-mon[54848]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-04-23T03:37:07.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:07 smithi149 ceph-mon[54848]: pgmap v185: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 127 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:07.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:37:07.336+0000 7f3e3f9e47c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-04-23T03:37:07.422 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:37:07.379+0000 7f3e3f9e47c0 -1 osd.7 140 log_to_monitors true 2024-04-23T03:37:07.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:07 smithi007 ceph-mon[64248]: pgmap v185: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 127 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:07.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:07 smithi007 ceph-mon[61772]: pgmap v185: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 0 B/s wr, 127 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:08.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:08.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[64248]: from='osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:37:08.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[64248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:37:08.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:08.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[61772]: from='osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:37:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[61772]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:37:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:08.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:08 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:08.675 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:08 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:08.675 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:08 smithi149 ceph-mon[54848]: from='osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:37:08.675 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:08 smithi149 ceph-mon[54848]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-23T03:37:08.675 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:08.675 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:08 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:09.171 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:37:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:37:08.764+0000 7f3e2ce34700 -1 osd.7 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-23T03:37:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[64248]: pgmap v186: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 169 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[64248]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T03:37:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[64248]: osdmap e143: 8 total, 7 up, 8 in 2024-04-23T03:37:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[64248]: from='osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:37:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[64248]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:37:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[64248]: from='osd.7 ' entity='osd.7' 2024-04-23T03:37:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[61772]: pgmap v186: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 169 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[61772]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T03:37:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[61772]: osdmap e143: 8 total, 7 up, 8 in 2024-04-23T03:37:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[61772]: from='osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:37:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[61772]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:37:09.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:09 smithi007 ceph-mon[61772]: from='osd.7 ' entity='osd.7' 2024-04-23T03:37:09.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:09 smithi149 ceph-mon[54848]: pgmap v186: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 246 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 169 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:09.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:09 smithi149 ceph-mon[54848]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-23T03:37:09.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:09 smithi149 ceph-mon[54848]: osdmap e143: 8 total, 7 up, 8 in 2024-04-23T03:37:09.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:09 smithi149 ceph-mon[54848]: from='osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:37:09.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:09 smithi149 ceph-mon[54848]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi149", "root=default"]}]: dispatch 2024-04-23T03:37:09.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:09 smithi149 ceph-mon[54848]: from='osd.7 ' entity='osd.7' 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100] boot 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: osdmap e144: 8 total, 8 up, 8 in 2024-04-23T03:37:10.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:10 smithi149 ceph-mon[54848]: Health check update: Degraded data redundancy: 96/726 objects degraded (13.223%), 27 pgs degraded (PG_DEGRADED) 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100] boot 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: osdmap e144: 8 total, 8 up, 8 in 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[64248]: Health check update: Degraded data redundancy: 96/726 objects degraded (13.223%), 27 pgs degraded (PG_DEGRADED) 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:10.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-04-23T03:37:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-23T03:37:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: osd.7 [v2:172.21.15.149:6824/339872100,v1:172.21.15.149:6825/339872100] boot 2024-04-23T03:37:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: osdmap e144: 8 total, 8 up, 8 in 2024-04-23T03:37:10.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:10 smithi007 ceph-mon[61772]: Health check update: Degraded data redundancy: 96/726 objects degraded (13.223%), 27 pgs degraded (PG_DEGRADED) 2024-04-23T03:37:11.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:11 smithi149 ceph-mon[54848]: pgmap v188: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 151 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:11.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:11 smithi149 ceph-mon[54848]: osdmap e145: 8 total, 8 up, 8 in 2024-04-23T03:37:11.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:11 smithi007 ceph-mon[64248]: pgmap v188: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 151 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:11.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:11 smithi007 ceph-mon[64248]: osdmap e145: 8 total, 8 up, 8 in 2024-04-23T03:37:11.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:11 smithi007 ceph-mon[61772]: pgmap v188: 161 pgs: 27 active+undersized+degraded, 42 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 0 B/s wr, 151 op/s; 96/726 objects degraded (13.223%) 2024-04-23T03:37:11.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:11 smithi007 ceph-mon[61772]: osdmap e145: 8 total, 8 up, 8 in 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: pgmap v191: 161 pgs: 7 peering, 22 active+undersized+degraded, 37 active+undersized, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s; 73/726 objects degraded (10.055%); 0 B/s, 1 objects/s recovering 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-23T03:37:13.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-23T03:37:13.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:13 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-23T03:37:13.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: pgmap v191: 161 pgs: 7 peering, 22 active+undersized+degraded, 37 active+undersized, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s; 73/726 objects degraded (10.055%); 0 B/s, 1 objects/s recovering 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-23T03:37:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-23T03:37:13.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: pgmap v191: 161 pgs: 7 peering, 22 active+undersized+degraded, 37 active+undersized, 95 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 63 op/s; 73/726 objects degraded (10.055%); 0 B/s, 1 objects/s recovering 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-23T03:37:13.835 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-04-23T03:37:13.836 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-23T03:37:13.837 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-23T03:37:13.838 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:13 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all osd 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: Upgrade: Setting require_osd_release to 19 squid 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: osdmap e146: 8 total, 8 up, 8 in 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:14.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:14 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:14.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all osd 2024-04-23T03:37:14.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: Upgrade: Setting require_osd_release to 19 squid 2024-04-23T03:37:14.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-23T03:37:14.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-23T03:37:14.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: osdmap e146: 8 total, 8 up, 8 in 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all osd 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: Upgrade: Setting require_osd_release to 19 squid 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: osdmap e146: 8 total, 8 up, 8 in 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:14 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:15.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:15 smithi149 ceph-mon[54848]: pgmap v192: 161 pgs: 7 peering, 6 active+undersized, 148 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 7 objects/s recovering 2024-04-23T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:15 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all mds 2024-04-23T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:15 smithi149 ceph-mon[54848]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/726 objects degraded (10.055%), 22 pgs degraded) 2024-04-23T03:37:15.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:15 smithi149 ceph-mon[54848]: Cluster is now healthy 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[64248]: pgmap v192: 161 pgs: 7 peering, 6 active+undersized, 148 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 7 objects/s recovering 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all mds 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[64248]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/726 objects degraded (10.055%), 22 pgs degraded) 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[64248]: Cluster is now healthy 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[61772]: pgmap v192: 161 pgs: 7 peering, 6 active+undersized, 148 active+clean; 585 KiB data, 261 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 7 objects/s recovering 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all mds 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[61772]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 73/726 objects degraded (10.055%), 22 pgs degraded) 2024-04-23T03:37:15.682 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:15 smithi007 ceph-mon[61772]: Cluster is now healthy 2024-04-23T03:37:16.383 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:37:16] "GET /metrics HTTP/1.1" 200 38553 "" "Prometheus/2.33.4" 2024-04-23T03:37:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-04-23T03:37:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: Upgrade: Updating rgw.foo.smithi007.bipufo (1/4) 2024-04-23T03:37:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:16.700 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[64248]: Deploying daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: Upgrade: Updating rgw.foo.smithi007.bipufo (1/4) 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:16.701 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:16 smithi007 ceph-mon[61772]: Deploying daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s; 0 B/s, 8 objects/s recovering 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: Upgrade: Updating rgw.foo.smithi007.bipufo (1/4) 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.bipufo", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:16 smithi149 ceph-mon[54848]: Deploying daemon rgw.foo.smithi007.bipufo on smithi007 2024-04-23T03:37:17.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:17 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:17.684 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:17 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:17.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:17 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:18.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:18 smithi007 ceph-mon[61772]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 263 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-04-23T03:37:18.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:18 smithi007 ceph-mon[64248]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 263 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-04-23T03:37:18.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:18 smithi149 ceph-mon[54848]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 263 B/s rd, 0 op/s; 0 B/s, 6 objects/s recovering 2024-04-23T03:37:21.053 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:37:21.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[61772]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-04-23T03:37:21.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[64248]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-04-23T03:37:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:21 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:21 smithi149 ceph-mon[54848]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 816 B/s rd, 0 op/s; 0 B/s, 5 objects/s recovering 2024-04-23T03:37:21.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:21 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:21 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:21 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (6m) 2m ago 11m 24.2M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (6m) 13s ago 11m 55.0M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (5m) 2m ago 10m 52.9M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (5m) 13s ago 15m 453M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (6m) 2m ago 17m 507M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (5m) 2m ago 17m 51.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (4m) 13s ago 16m 39.7M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (4m) 2m ago 16m 44.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (6m) 2m ago 11m 17.1M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (6m) 13s ago 11m 17.2M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (3m) 2m ago 15m 44.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:37:21.762 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (3m) 2m ago 15m 43.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (2m) 2m ago 14m 44.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (2m) 2m ago 14m 12.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (119s) 13s ago 13m 50.6M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (86s) 13s ago 13m 46.5M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 f88b3de0b1f7 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (52s) 13s ago 13m 55.9M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 ec30c8d5356b 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (17s) 13s ago 12m 14.9M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 7a072e4b062d 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (5m) 13s ago 12m 76.2M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 starting - - - - 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (10m) 13s ago 10m 77.0M - 17.2.0 e1d6a67b021e cd06e58d99af 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (10m) 2m ago 10m 75.8M - 17.2.0 e1d6a67b021e a5cd9ed9eccb 2024-04-23T03:37:21.763 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (10m) 13s ago 10m 76.3M - 17.2.0 e1d6a67b021e d87c6b025483 2024-04-23T03:37:22.217 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:37:22.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:37:22.217 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:37:22.217 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:22.217 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:37:22.217 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 8 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 1 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 14 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:37:22.218 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:37:22.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[64248]: from='client.44526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:22.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:22.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:22.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:22.357 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:22.357 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[61772]: from='client.44526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:22.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:22.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:22.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:22.358 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:22 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:22.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:22 smithi149 ceph-mon[54848]: from='client.44526 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:22.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:22 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.mxntqi", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:22.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:22 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:22.580 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:37:22.580 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:37:22.580 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:37:22.580 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "osd" 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "13/23 daemons upgraded", 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading rgw daemons", 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:37:22.581 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:37:23.005 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s; 0 B/s, 4 objects/s recovering 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: Upgrade: Updating rgw.smpl.smithi007.mxntqi (2/4) 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: Deploying daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3668765269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/4264073995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:37:23.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s; 0 B/s, 4 objects/s recovering 2024-04-23T03:37:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: Upgrade: Updating rgw.smpl.smithi007.mxntqi (2/4) 2024-04-23T03:37:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: Deploying daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:37:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3668765269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:23.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:23 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/4264073995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:37:23.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: from='client.44532 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:23.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 28 op/s; 0 B/s, 4 objects/s recovering 2024-04-23T03:37:23.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: from='client.54469 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:23.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: Upgrade: Updating rgw.smpl.smithi007.mxntqi (2/4) 2024-04-23T03:37:23.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: Deploying daemon rgw.smpl.smithi007.mxntqi on smithi007 2024-04-23T03:37:23.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3668765269' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:23.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:23 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/4264073995' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:37:24.315 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:24 smithi007 ceph-mon[64248]: from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:24.315 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:24 smithi007 ceph-mon[61772]: from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:24.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:24 smithi149 ceph-mon[54848]: from='client.44550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:25.208 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:25 smithi007 ceph-mon[61772]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 102 B/s wr, 126 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T03:37:25.208 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:25 smithi007 ceph-mon[64248]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 102 B/s wr, 126 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T03:37:25.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:25 smithi149 ceph-mon[54848]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 102 B/s wr, 126 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T03:37:26.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:37:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:37:26] "GET /metrics HTTP/1.1" 200 38553 "" "Prometheus/2.33.4" 2024-04-23T03:37:27.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:27 smithi149 ceph-mon[54848]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 87 B/s wr, 126 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:27 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:27.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:27 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:27.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:27 smithi007 ceph-mon[64248]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 87 B/s wr, 126 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T03:37:27.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:27 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:27.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:27 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:27 smithi007 ceph-mon[61772]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 80 KiB/s rd, 87 B/s wr, 126 op/s; 0 B/s, 0 objects/s recovering 2024-04-23T03:37:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:27 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:27.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:27 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:28 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:28 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:28 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:28 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:28.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:28 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi149.axrzpl", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:28 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:29.359 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:29 smithi149 ceph-mon[54848]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 180 op/s 2024-04-23T03:37:29.359 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:29 smithi149 ceph-mon[54848]: Upgrade: Updating rgw.foo.smithi149.axrzpl (3/4) 2024-04-23T03:37:29.359 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:29 smithi149 ceph-mon[54848]: Deploying daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:37:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:29 smithi007 ceph-mon[64248]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 180 op/s 2024-04-23T03:37:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:29 smithi007 ceph-mon[64248]: Upgrade: Updating rgw.foo.smithi149.axrzpl (3/4) 2024-04-23T03:37:29.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:29 smithi007 ceph-mon[64248]: Deploying daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:37:29.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:29 smithi007 ceph-mon[61772]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 180 op/s 2024-04-23T03:37:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:29 smithi007 ceph-mon[61772]: Upgrade: Updating rgw.foo.smithi149.axrzpl (3/4) 2024-04-23T03:37:29.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:29 smithi007 ceph-mon[61772]: Deploying daemon rgw.foo.smithi149.axrzpl on smithi149 2024-04-23T03:37:31.400 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:31 smithi149 ceph-mon[54848]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 170 B/s wr, 230 op/s 2024-04-23T03:37:31.400 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:31 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:31.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:31 smithi007 ceph-mon[61772]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 170 B/s wr, 230 op/s 2024-04-23T03:37:31.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:31 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:31 smithi007 ceph-mon[64248]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 146 KiB/s rd, 170 B/s wr, 230 op/s 2024-04-23T03:37:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:31 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:32.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:32 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:32.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:32 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:32.739 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:32 smithi149 ceph-mon[54848]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:32 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:32.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:32 smithi007 ceph-mon[64248]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:32 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:32.834 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:32 smithi007 ceph-mon[61772]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:34.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:34 smithi149 ceph-mon[54848]: Upgrade: Updating rgw.smpl.smithi149.rtdeuy (4/4) 2024-04-23T03:37:34.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:34 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:34.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:34 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:34.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:34 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:34.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:34 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:34.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:34 smithi149 ceph-mon[54848]: Deploying daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:37:34.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[64248]: Upgrade: Updating rgw.smpl.smithi149.rtdeuy (4/4) 2024-04-23T03:37:34.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[64248]: Deploying daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[61772]: Upgrade: Updating rgw.smpl.smithi149.rtdeuy (4/4) 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi149.rtdeuy", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:34 smithi007 ceph-mon[61772]: Deploying daemon rgw.smpl.smithi149.rtdeuy on smithi149 2024-04-23T03:37:35.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:35 smithi149 ceph-mon[54848]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 255 B/s wr, 306 op/s 2024-04-23T03:37:35.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:35 smithi007 ceph-mon[64248]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 255 B/s wr, 306 op/s 2024-04-23T03:37:35.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:35 smithi007 ceph-mon[61772]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 273 MiB used, 715 GiB / 715 GiB avail; 194 KiB/s rd, 255 B/s wr, 306 op/s 2024-04-23T03:37:36.483 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:37:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:37:36] "GET /metrics HTTP/1.1" 200 38546 "" "Prometheus/2.33.4" 2024-04-23T03:37:37.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:37 smithi007 ceph-mon[64248]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 156 KiB/s rd, 170 B/s wr, 247 op/s 2024-04-23T03:37:37.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:37.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:37 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:37 smithi007 ceph-mon[61772]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 156 KiB/s rd, 170 B/s wr, 247 op/s 2024-04-23T03:37:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:37.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:37 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:37.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:37 smithi149 ceph-mon[54848]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 277 MiB used, 715 GiB / 715 GiB avail; 156 KiB/s rd, 170 B/s wr, 247 op/s 2024-04-23T03:37:37.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:37.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:37 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:38.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:38 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:38.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:38 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:38.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:38 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:39.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:39 smithi007 ceph-mon[64248]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:39.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:39 smithi007 ceph-mon[61772]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:39.640 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:39 smithi149 ceph-mon[54848]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:41.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:41 smithi007 ceph-mon[64248]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 85 B/s wr, 241 op/s 2024-04-23T03:37:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:41 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:41.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:41 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:41 smithi007 ceph-mon[61772]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 85 B/s wr, 241 op/s 2024-04-23T03:37:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:41 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:41.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:41 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:41.573 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:41 smithi149 ceph-mon[54848]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 85 B/s wr, 241 op/s 2024-04-23T03:37:41.573 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:41 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:41.574 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:41 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:42 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:42 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:42 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.496 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:42 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:42.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:42 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:43.290 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:43 smithi007 ceph-mon[64248]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 85 B/s wr, 227 op/s 2024-04-23T03:37:43.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:43 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:43.291 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:43 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:43.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:43 smithi007 ceph-mon[61772]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 85 B/s wr, 227 op/s 2024-04-23T03:37:43.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:43 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:43.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:43 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:43.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:43 smithi149 ceph-mon[54848]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s rd, 85 B/s wr, 227 op/s 2024-04-23T03:37:43.499 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:43 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:43.500 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:43 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:45.384 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:45 smithi149 ceph-mon[54848]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:45.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:45 smithi007 ceph-mon[61772]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:45.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:45 smithi007 ceph-mon[64248]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-04-23T03:37:46.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:37:46] "GET /metrics HTTP/1.1" 200 38612 "" "Prometheus/2.33.4" 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: Detected new or changed devices on smithi149 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 85 B/s wr, 166 op/s 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: Detected new or changed devices on smithi007 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:37:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all rgw 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]': finished 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]: dispatch 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]': finished 2024-04-23T03:37:46.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]': finished 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]': finished 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all rbd-mirror 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: Upgrade: Setting container_image for all ceph-exporter 2024-04-23T03:37:46.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:46 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: Detected new or changed devices on smithi149 2024-04-23T03:37:46.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.724 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 85 B/s wr, 166 op/s 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: Detected new or changed devices on smithi007 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.725 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all rgw 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]: dispatch 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]: dispatch 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]': finished 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]: dispatch 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]: dispatch 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]': finished 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]: dispatch 2024-04-23T03:37:46.726 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]: dispatch 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: Detected new or changed devices on smithi149 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s rd, 85 B/s wr, 166 op/s 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: Detected new or changed devices on smithi007 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.727 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all rgw 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi007.bipufo"}]': finished 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]: dispatch 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi149.axrzpl"}]': finished 2024-04-23T03:37:46.728 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]': finished 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]': finished 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all rbd-mirror 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: Upgrade: Setting container_image for all ceph-exporter 2024-04-23T03:37:46.729 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi007.mxntqi"}]': finished 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]: dispatch 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]: dispatch 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi149.rtdeuy"}]': finished 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all rbd-mirror 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: Upgrade: Setting container_image for all ceph-exporter 2024-04-23T03:37:46.730 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:46 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:47.817 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:47 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:47.818 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:47 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:47.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:47 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:49.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 143 op/s 2024-04-23T03:37:49.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: Upgrade: Updating iscsi.foo.smithi007.xwmqjz 2024-04-23T03:37:49.278 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[64248]: Deploying daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 143 op/s 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: Upgrade: Updating iscsi.foo.smithi007.xwmqjz 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:49.279 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:48 smithi007 ceph-mon[61772]: Deploying daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:37:49.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 85 B/s wr, 143 op/s 2024-04-23T03:37:49.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: Upgrade: Updating iscsi.foo.smithi007.xwmqjz 2024-04-23T03:37:49.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:37:49.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:37:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.xwmqjz", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-23T03:37:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:37:49.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:48 smithi149 ceph-mon[54848]: Deploying daemon iscsi.foo.smithi007.xwmqjz on smithi007 2024-04-23T03:37:51.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:51 smithi007 ceph-mon[61772]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-04-23T03:37:51.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:51 smithi007 ceph-mon[64248]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-04-23T03:37:51.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:51 smithi149 ceph-mon[54848]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-04-23T03:37:53.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:53 smithi007 ceph-mon[64248]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 85 B/s wr, 75 op/s 2024-04-23T03:37:53.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:53 smithi007 ceph-mon[61772]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 85 B/s wr, 75 op/s 2024-04-23T03:37:53.383 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:37:53.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:53 smithi149 ceph-mon[54848]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 48 KiB/s rd, 85 B/s wr, 75 op/s 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (6m) 12s ago 11m 24.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (6m) 13s ago 11m 55.1M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 running (6m) 12s ago 10m 58.9M - 3.5 e1d6a67b021e cc9bd04863ff 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (6m) 13s ago 16m 453M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (6m) 12s ago 17m 512M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (5m) 12s ago 18m 54.0M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (5m) 13s ago 16m 41.8M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:37:54.065 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (5m) 12s ago 16m 46.1M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (7m) 12s ago 12m 20.5M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (6m) 13s ago 12m 17.2M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (4m) 12s ago 15m 55.6M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (3m) 12s ago 15m 57.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (3m) 12s ago 15m 53.8M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (3m) 12s ago 14m 65.8M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (2m) 13s ago 14m 57.7M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (118s) 13s ago 14m 50.7M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 f88b3de0b1f7 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (84s) 13s ago 13m 62.1M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 ec30c8d5356b 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (50s) 13s ago 13m 61.5M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 7a072e4b062d 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (6m) 13s ago 12m 76.2M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (34s) 12s ago 11m 76.2M - 19.0.0-3198-g56e81df3 82086b19bb07 2460b2288b6e 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (23s) 13s ago 11m 77.5M - 19.0.0-3198-g56e81df3 82086b19bb07 6d2a2f0efe30 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (28s) 12s ago 10m 76.1M - 19.0.0-3198-g56e81df3 82086b19bb07 fcea1228ccfd 2024-04-23T03:37:54.066 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (17s) 13s ago 10m 75.1M - 19.0.0-3198-g56e81df3 82086b19bb07 ebb99282cbed 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 8 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:54.481 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:37:54.482 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 4 2024-04-23T03:37:54.482 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:37:54.482 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:37:54.482 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 17 2024-04-23T03:37:54.482 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:37:54.482 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:37:54.828 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:37:54.828 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:37:54.828 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:37:54.828 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:37:54.828 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw", 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "osd" 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "17/23 daemons upgraded", 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Currently upgrading iscsi daemons", 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": false 2024-04-23T03:37:54.829 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:37:55.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[61772]: from='client.44625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[61772]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 85 B/s wr, 39 op/s 2024-04-23T03:37:55.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[61772]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[61772]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/32695329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[64248]: from='client.44625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[64248]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 85 B/s wr, 39 op/s 2024-04-23T03:37:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[64248]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[64248]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:55 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/32695329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:55.246 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_OK 2024-04-23T03:37:55.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:55 smithi149 ceph-mon[54848]: from='client.44625 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:55 smithi149 ceph-mon[54848]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 85 B/s wr, 39 op/s 2024-04-23T03:37:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:55 smithi149 ceph-mon[54848]: from='client.54559 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:55 smithi149 ceph-mon[54848]: from='client.54565 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:55.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:55 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/32695329' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:37:56.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:56 smithi007 ceph-mon[61772]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:56.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:56 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3879662127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:37:56.332 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:37:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:37:56] "GET /metrics HTTP/1.1" 200 38612 "" "Prometheus/2.33.4" 2024-04-23T03:37:56.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:56 smithi007 ceph-mon[64248]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:56.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:56 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3879662127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:37:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:56 smithi149 ceph-mon[54848]: from='client.44649 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:37:56.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:56 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3879662127' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:37:57.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:57 smithi007 ceph-mon[61772]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:37:57.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:57 smithi007 ceph-mon[64248]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:37:57.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:57 smithi149 ceph-mon[54848]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:37:58.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:58 smithi007 ceph-mon[61772]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:58.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:58 smithi007 ceph-mon[64248]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:58.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:58 smithi149 ceph-mon[54848]: from='client.25294 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:37:59.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:37:59 smithi007 ceph-mon[61772]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:37:59.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:37:59 smithi007 ceph-mon[64248]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:37:59.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:37:59 smithi149 ceph-mon[54848]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:01.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:01 smithi007 ceph-mon[61772]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:01.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:01.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:01 smithi007 ceph-mon[64248]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:01.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:01 smithi149 ceph-mon[54848]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:01.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:01.832 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:01 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:38:01.457+0000 7f1c406a1700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:38:02.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:02 smithi007 ceph-mon[61772]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:38:02.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:02 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:02.498 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:02 smithi007 ceph-mon[61772]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-23T03:38:02.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:02 smithi007 ceph-mon[64248]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:38:02.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:02 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:02.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:02 smithi007 ceph-mon[64248]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-23T03:38:02.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:02 smithi149 ceph-mon[54848]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:38:02.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:02 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:02.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:02 smithi149 ceph-mon[54848]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 859 B/s rd, 0 op/s 2024-04-23T03:38:03.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:03 smithi007 ceph-mon[64248]: Health check failed: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-23T03:38:03.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:03 smithi007 ceph-mon[61772]: Health check failed: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-23T03:38:03.841 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:03 smithi149 ceph-mon[54848]: Health check failed: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-04-23T03:38:04.561 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:04 smithi007 ceph-mon[64248]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:38:04.561 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:04 smithi007 ceph-mon[61772]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:38:04.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:04 smithi149 ceph-mon[54848]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:38:06.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:06.172 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:05 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:06.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:06.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:05 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:06.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:06.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:05 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:06.333 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:38:06] "GET /metrics HTTP/1.1" 200 38610 "" "Prometheus/2.33.4" 2024-04-23T03:38:07.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2024-04-23T03:38:07.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:38:07.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:38:07.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: Checking dashboard <-> RGW credentials 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-04-23T03:38:07.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: Checking dashboard <-> RGW credentials 2024-04-23T03:38:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:38:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:38:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-23T03:38:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-04-23T03:38:07.334 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:06 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 687 B/s rd, 0 op/s 2024-04-23T03:38:07.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 787 B/s rd, 0 op/s 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: Checking dashboard <-> RGW credentials 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 934 B/s rd, 0 op/s 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 454 B/s rd, 0 op/s 2024-04-23T03:38:07.422 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:06 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:08.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:07 smithi007 ceph-mon[64248]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:38:08.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:07 smithi007 ceph-mon[61772]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:38:08.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:07 smithi149 ceph-mon[54848]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:38:09.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:08 smithi007 ceph-mon[64248]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-04-23T03:38:09.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:08 smithi007 ceph-mon[61772]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-04-23T03:38:09.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:08 smithi149 ceph-mon[54848]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-04-23T03:38:10.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:10 smithi149 ceph-mon[54848]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-23T03:38:10.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:10 smithi007 ceph-mon[64248]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-23T03:38:10.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:10 smithi007 ceph-mon[61772]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-23T03:38:12.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:12 smithi149 ceph-mon[54848]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-23T03:38:12.714 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:12 smithi007 ceph-mon[61772]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-23T03:38:12.715 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:12 smithi007 ceph-mon[64248]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 3 op/s 2024-04-23T03:38:13.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:13 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3592138589' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:38:13.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:13 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2636546025' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]: dispatch 2024-04-23T03:38:13.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:13 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]: dispatch 2024-04-23T03:38:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:13 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3592138589' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:38:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:13 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2636546025' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]: dispatch 2024-04-23T03:38:13.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:13 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]: dispatch 2024-04-23T03:38:13.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:13 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3592138589' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-23T03:38:13.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:13 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2636546025' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]: dispatch 2024-04-23T03:38:13.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:13 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]: dispatch 2024-04-23T03:38:14.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]': finished 2024-04-23T03:38:14.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[64248]: osdmap e147: 8 total, 8 up, 8 in 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/818933198' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]: dispatch 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]: dispatch 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[64248]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]': finished 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[61772]: osdmap e147: 8 total, 8 up, 8 in 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/818933198' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]: dispatch 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]: dispatch 2024-04-23T03:38:14.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:14 smithi007 ceph-mon[61772]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-04-23T03:38:14.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:14 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2921287867"}]': finished 2024-04-23T03:38:14.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:14 smithi149 ceph-mon[54848]: osdmap e147: 8 total, 8 up, 8 in 2024-04-23T03:38:14.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:14 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/818933198' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]: dispatch 2024-04-23T03:38:14.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:14 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]: dispatch 2024-04-23T03:38:14.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:14 smithi149 ceph-mon[54848]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-04-23T03:38:15.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]': finished 2024-04-23T03:38:15.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[61772]: osdmap e148: 8 total, 8 up, 8 in 2024-04-23T03:38:15.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2770657556' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]: dispatch 2024-04-23T03:38:15.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]: dispatch 2024-04-23T03:38:15.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]': finished 2024-04-23T03:38:15.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[64248]: osdmap e148: 8 total, 8 up, 8 in 2024-04-23T03:38:15.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2770657556' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]: dispatch 2024-04-23T03:38:15.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:15 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]: dispatch 2024-04-23T03:38:15.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:15 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1724366908"}]': finished 2024-04-23T03:38:15.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:15 smithi149 ceph-mon[54848]: osdmap e148: 8 total, 8 up, 8 in 2024-04-23T03:38:15.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:15 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2770657556' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]: dispatch 2024-04-23T03:38:15.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:15 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]: dispatch 2024-04-23T03:38:16.397 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:38:16] "GET /metrics HTTP/1.1" 200 38690 "" "Prometheus/2.33.4" 2024-04-23T03:38:16.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]': finished 2024-04-23T03:38:16.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: osdmap e149: 8 total, 8 up, 8 in 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3668268063' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]: dispatch 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]: dispatch 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[61772]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]': finished 2024-04-23T03:38:16.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: osdmap e149: 8 total, 8 up, 8 in 2024-04-23T03:38:16.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:16.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:16.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3668268063' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]: dispatch 2024-04-23T03:38:16.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]: dispatch 2024-04-23T03:38:16.834 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:16 smithi007 ceph-mon[64248]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1724366908"}]': finished 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: osdmap e149: 8 total, 8 up, 8 in 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3668268063' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]: dispatch 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]: dispatch 2024-04-23T03:38:16.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:16 smithi149 ceph-mon[54848]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-04-23T03:38:17.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:17 smithi007 ceph-mon[64248]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]': finished 2024-04-23T03:38:17.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:17 smithi007 ceph-mon[64248]: osdmap e150: 8 total, 8 up, 8 in 2024-04-23T03:38:17.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:17 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/1425739768' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3089481092"}]: dispatch 2024-04-23T03:38:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:17 smithi007 ceph-mon[61772]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]': finished 2024-04-23T03:38:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:17 smithi007 ceph-mon[61772]: osdmap e150: 8 total, 8 up, 8 in 2024-04-23T03:38:17.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:17 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/1425739768' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3089481092"}]: dispatch 2024-04-23T03:38:17.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:17 smithi149 ceph-mon[54848]: from='client.? ' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3323500973"}]': finished 2024-04-23T03:38:17.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:17 smithi149 ceph-mon[54848]: osdmap e150: 8 total, 8 up, 8 in 2024-04-23T03:38:17.923 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:17 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/1425739768' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3089481092"}]: dispatch 2024-04-23T03:38:18.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/1425739768' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3089481092"}]': finished 2024-04-23T03:38:18.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[61772]: osdmap e151: 8 total, 8 up, 8 in 2024-04-23T03:38:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/1641987100' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1895491992"}]: dispatch 2024-04-23T03:38:18.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[61772]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:18.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/1425739768' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3089481092"}]': finished 2024-04-23T03:38:18.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[64248]: osdmap e151: 8 total, 8 up, 8 in 2024-04-23T03:38:18.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/1641987100' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1895491992"}]: dispatch 2024-04-23T03:38:18.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:18 smithi007 ceph-mon[64248]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:18.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:18 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/1425739768' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3089481092"}]': finished 2024-04-23T03:38:18.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:18 smithi149 ceph-mon[54848]: osdmap e151: 8 total, 8 up, 8 in 2024-04-23T03:38:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:18 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/1641987100' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1895491992"}]: dispatch 2024-04-23T03:38:18.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:18 smithi149 ceph-mon[54848]: pgmap v233: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:19.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:19 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/1641987100' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1895491992"}]': finished 2024-04-23T03:38:19.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:19 smithi007 ceph-mon[61772]: osdmap e152: 8 total, 8 up, 8 in 2024-04-23T03:38:19.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:19 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/1641987100' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1895491992"}]': finished 2024-04-23T03:38:19.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:19 smithi007 ceph-mon[64248]: osdmap e152: 8 total, 8 up, 8 in 2024-04-23T03:38:19.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:19 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/1641987100' entity='client.iscsi.foo.smithi007.xwmqjz' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1895491992"}]': finished 2024-04-23T03:38:19.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:19 smithi149 ceph-mon[54848]: osdmap e152: 8 total, 8 up, 8 in 2024-04-23T03:38:20.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:20 smithi007 ceph-mon[61772]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:38:20.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:20 smithi007 ceph-mon[64248]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:38:20.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:20 smithi149 ceph-mon[54848]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-23T03:38:22.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:22 smithi149 ceph-mon[54848]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:22.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:22 smithi007 ceph-mon[61772]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:22.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:22 smithi007 ceph-mon[64248]: pgmap v236: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:23.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:23 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:23.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:23 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:23.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:23 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:24.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:24 smithi007 ceph-mon[64248]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:38:24.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:24 smithi007 ceph-mon[61772]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:38:24.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:24 smithi149 ceph-mon[54848]: pgmap v237: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-23T03:38:25.629 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:38:25.981 INFO:teuthology.orchestra.run.smithi007.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed." 2024-04-23T03:38:26.315 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:38:26] "GET /metrics HTTP/1.1" 200 38690 "" "Prometheus/2.33.4" 2024-04-23T03:38:26.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-mon[64248]: from='client.44757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:26.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-mon[64248]: from='client.54691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:26.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-mon[64248]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:38:26.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-mon[61772]: from='client.44757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:26.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-mon[61772]: from='client.54691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:26.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:26 smithi007 ceph-mon[61772]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:38:26.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:26 smithi149 ceph-mon[54848]: from='client.44757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:26.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:26 smithi149 ceph-mon[54848]: from='client.54691 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:26.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:26 smithi149 ceph-mon[54848]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:38:26.726 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch ps' 2024-04-23T03:38:28.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:28 smithi007 ceph-mon[64248]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:38:28.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:28 smithi007 ceph-mon[61772]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:38:28.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:28 smithi149 ceph-mon[54848]: pgmap v239: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:38:28.794 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (7m) 22s ago 12m 24.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (7m) 23s ago 12m 55.4M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 unknown 22s ago 11m - - 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (6m) 23s ago 16m 453M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (7m) 22s ago 18m 513M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (6m) 22s ago 18m 55.0M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (5m) 23s ago 17m 43.0M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (5m) 22s ago 17m 47.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:38:28.795 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (7m) 22s ago 13m 17.5M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (7m) 23s ago 13m 16.8M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (4m) 22s ago 16m 55.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (4m) 22s ago 16m 57.5M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (4m) 22s ago 15m 54.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (3m) 22s ago 15m 65.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (3m) 23s ago 15m 57.9M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (2m) 23s ago 14m 51.0M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 f88b3de0b1f7 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (119s) 23s ago 14m 62.2M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 ec30c8d5356b 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (85s) 23s ago 13m 61.8M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 7a072e4b062d 2024-04-23T03:38:28.796 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (6m) 23s ago 13m 76.3M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:38:28.797 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (69s) 22s ago 11m 76.6M - 19.0.0-3198-g56e81df3 82086b19bb07 2460b2288b6e 2024-04-23T03:38:28.797 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (57s) 23s ago 11m 77.7M - 19.0.0-3198-g56e81df3 82086b19bb07 6d2a2f0efe30 2024-04-23T03:38:28.797 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (63s) 22s ago 11m 76.4M - 19.0.0-3198-g56e81df3 82086b19bb07 fcea1228ccfd 2024-04-23T03:38:28.797 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (52s) 23s ago 11m 75.5M - 19.0.0-3198-g56e81df3 82086b19bb07 ebb99282cbed 2024-04-23T03:38:29.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph versions' 2024-04-23T03:38:29.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:29 smithi149 ceph-mon[54848]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:29.820 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:29 smithi007 ceph-mon[64248]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:29.820 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:29 smithi007 ceph-mon[61772]: from='client.44769 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:38:30.813 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:30 smithi007 ceph-mon[64248]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-04-23T03:38:30.813 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:30 smithi007 ceph-mon[61772]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-04-23T03:38:30.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:30 smithi149 ceph-mon[54848]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 870 B/s rd, 0 op/s 2024-04-23T03:38:31.461 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:38:31.461 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:38:31.461 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:38:31.461 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 8 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:38:31.462 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 4 2024-04-23T03:38:31.463 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:38:31.463 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:38:31.463 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 17 2024-04-23T03:38:31.463 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:38:31.463 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:38:31.829 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:31 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:31.830 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:31 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:31.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:31 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:32.176 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-23T03:38:32.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:32 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/3003115888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:38:32.736 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:32 smithi007 ceph-mon[64248]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:32.736 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:32 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/3003115888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:38:32.737 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:32 smithi007 ceph-mon[61772]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:32.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:32 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/3003115888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:38:32.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:32 smithi149 ceph-mon[54848]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:33.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:33 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:33.686 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:33 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:33.885 INFO:teuthology.orchestra.run.smithi007.stdout:wait for servicemap items w/ changing names to refresh 2024-04-23T03:38:33.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:33 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:34.523 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:34 smithi007 ceph-mon[64248]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:34.524 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:34 smithi007 ceph-mon[61772]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:34.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'sleep 60' 2024-04-23T03:38:34.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:34 smithi149 ceph-mon[54848]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:36.385 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:38:36] "GET /metrics HTTP/1.1" 200 38688 "" "Prometheus/2.33.4" 2024-04-23T03:38:36.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:36 smithi149 ceph-mon[54848]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:36.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:36 smithi007 ceph-mon[64248]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:36.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:36 smithi007 ceph-mon[61772]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:38.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:38 smithi149 ceph-mon[54848]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:38.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:38 smithi007 ceph-mon[61772]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:38.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:38 smithi007 ceph-mon[64248]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:40.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:40 smithi149 ceph-mon[54848]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:40.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:40 smithi007 ceph-mon[61772]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:40.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:40 smithi007 ceph-mon[64248]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:42.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:42 smithi149 ceph-mon[54848]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:42.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:42 smithi007 ceph-mon[61772]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:42.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:42 smithi007 ceph-mon[64248]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:43.725 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:43 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:43.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:43 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:43.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:43 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:44.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:44 smithi007 ceph-mon[64248]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:44.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:44 smithi007 ceph-mon[61772]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:44.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:44 smithi149 ceph-mon[54848]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:45.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:45 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:46.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:45 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:46.083 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:45 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:38:46.582 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:38:46] "GET /metrics HTTP/1.1" 200 38691 "" "Prometheus/2.33.4" 2024-04-23T03:38:46.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:46 smithi149 ceph-mon[54848]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:47.082 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:46 smithi007 ceph-mon[64248]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:47.082 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:46 smithi007 ceph-mon[61772]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:48.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:48 smithi149 ceph-mon[54848]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:48.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:48 smithi007 ceph-mon[61772]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:48.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:48 smithi007 ceph-mon[64248]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:50.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:50 smithi149 ceph-mon[54848]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:50.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:50 smithi007 ceph-mon[64248]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:50.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:50 smithi007 ceph-mon[61772]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:52.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:52 smithi007 ceph-mon[64248]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:52.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:52 smithi007 ceph-mon[61772]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:52.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:52 smithi149 ceph-mon[54848]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:53.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:53 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:53.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:53 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:53.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:53 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:38:54.713 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:54 smithi149 ceph-mon[54848]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:54.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:54 smithi007 ceph-mon[64248]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:54.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:54 smithi007 ceph-mon[61772]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:56.422 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:38:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:38:56] "GET /metrics HTTP/1.1" 200 38691 "" "Prometheus/2.33.4" 2024-04-23T03:38:56.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:56 smithi007 ceph-mon[64248]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:56.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:56 smithi007 ceph-mon[61772]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:56.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:56 smithi149 ceph-mon[54848]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:38:58.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:38:58 smithi007 ceph-mon[64248]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:58.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:38:58 smithi007 ceph-mon[61772]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:38:58.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:38:58 smithi149 ceph-mon[54848]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:00.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:00 smithi007 ceph-mon[64248]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:00.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:00 smithi007 ceph-mon[61772]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:00.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:00 smithi149 ceph-mon[54848]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:01.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:01.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:01.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:02.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:02 smithi007 ceph-mon[64248]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:02.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:02 smithi007 ceph-mon[61772]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:02.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:02 smithi149 ceph-mon[54848]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:03.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:03 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:03.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:03 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:03.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:03 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:04.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:04 smithi007 ceph-mon[64248]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:04.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:04 smithi007 ceph-mon[61772]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:04.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:04 smithi149 ceph-mon[54848]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:06.424 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:39:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:39:06] "GET /metrics HTTP/1.1" 200 38690 "" "Prometheus/2.33.4" 2024-04-23T03:39:06.693 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:06 smithi007 ceph-mon[64248]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:06.693 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:06 smithi007 ceph-mon[61772]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:06.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:06 smithi149 ceph-mon[54848]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:07.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:39:07.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:39:07.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[61772]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-04-23T03:39:07.832 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[61772]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 582 B/s rd, 0 op/s 2024-04-23T03:39:07.833 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[61772]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:39:07.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:39:07.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:39:07.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[64248]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-04-23T03:39:07.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[64248]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 582 B/s rd, 0 op/s 2024-04-23T03:39:07.833 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:07 smithi007 ceph-mon[64248]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:39:07.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:07 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-23T03:39:07.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:07 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-23T03:39:07.921 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:07 smithi149 ceph-mon[54848]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 948 B/s rd, 0 op/s 2024-04-23T03:39:07.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:07 smithi149 ceph-mon[54848]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 582 B/s rd, 0 op/s 2024-04-23T03:39:07.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:07 smithi149 ceph-mon[54848]: from='mgr.25354 ' entity='mgr.y' 2024-04-23T03:39:10.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:10 smithi007 ceph-mon[64248]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:10.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:10 smithi007 ceph-mon[61772]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:10.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:10 smithi149 ceph-mon[54848]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:12.474 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:12 smithi149 ceph-mon[54848]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:12.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:12 smithi007 ceph-mon[61772]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:12.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:12 smithi007 ceph-mon[64248]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:14.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:14 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:14.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:14 smithi007 ceph-mon[61772]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:14 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:14.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:14 smithi007 ceph-mon[64248]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:14.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:14 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:14.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:14 smithi149 ceph-mon[54848]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:16.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:16 smithi007 ceph-mon[64248]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:16.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:16 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:16 smithi007 ceph-mon[61772]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:16.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:16 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:16.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:39:16 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:39:16] "GET /metrics HTTP/1.1" 200 38687 "" "Prometheus/2.33.4" 2024-04-23T03:39:16.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:16 smithi149 ceph-mon[54848]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-23T03:39:16.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:16 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:18.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:18 smithi007 ceph-mon[64248]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:39:18.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:18 smithi007 ceph-mon[61772]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:39:18.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:18 smithi149 ceph-mon[54848]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-23T03:39:20.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:20 smithi007 ceph-mon[64248]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:20.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:20 smithi007 ceph-mon[61772]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:20.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:20 smithi149 ceph-mon[54848]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:22.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:22 smithi007 ceph-mon[64248]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:22.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:22 smithi007 ceph-mon[61772]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:22.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:22 smithi149 ceph-mon[54848]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:24.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:24 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:24.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:24 smithi007 ceph-mon[64248]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:24 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:24.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:24 smithi007 ceph-mon[61772]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:24.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:24 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:24.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:24 smithi149 ceph-mon[54848]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:26.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:26 smithi007 ceph-mon[61772]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:39:26.583 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:39:26 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:39:26] "GET /metrics HTTP/1.1" 200 38687 "" "Prometheus/2.33.4" 2024-04-23T03:39:26.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:26 smithi007 ceph-mon[64248]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:39:26.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:26 smithi149 ceph-mon[54848]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:39:28.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:28 smithi007 ceph-mon[61772]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:39:28.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:28 smithi007 ceph-mon[64248]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:39:28.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:28 smithi149 ceph-mon[54848]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-23T03:39:30.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:30 smithi007 ceph-mon[61772]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:30.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:30 smithi007 ceph-mon[64248]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:30.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:30 smithi149 ceph-mon[54848]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:31.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:31 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:31.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:31 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:31.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:31 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:32.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:32 smithi007 ceph-mon[64248]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:32.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:32 smithi007 ceph-mon[61772]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:32.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:32 smithi149 ceph-mon[54848]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:34.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:34 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:34.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:34 smithi007 ceph-mon[61772]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:34 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:34.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:34 smithi007 ceph-mon[64248]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:34.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:34 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:34.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:34 smithi149 ceph-mon[54848]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:36.384 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:36 smithi007 ceph-mon[64248]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:36.385 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:39:36 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:39:36] "GET /metrics HTTP/1.1" 200 38693 "" "Prometheus/2.33.4" 2024-04-23T03:39:36.385 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:36 smithi007 ceph-mon[61772]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:36.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:36 smithi149 ceph-mon[54848]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:37.193 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch ps' 2024-04-23T03:39:38.536 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:38 smithi007 ceph-mon[61772]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:38.537 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:38 smithi007 ceph-mon[64248]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:38.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:38 smithi149 ceph-mon[54848]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:39.422 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (8m) 93s ago 13m 24.4M - 0.23.0 ba2b418f427c dd67074f48e4 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi149 *:3000 running (8m) 94s ago 13m 55.4M - 8.3.5 dad864ee21e9 4865bb18d5e1 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.xwmqjz smithi007 unknown 93s ago 12m - - 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi149 *:8443,9283,8765 running (7m) 94s ago 18m 453M - 19.0.0-3198-g56e81df3 82086b19bb07 d525caf93187 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:8443,9283,8765 running (8m) 93s ago 19m 513M - 19.0.0-3198-g56e81df3 82086b19bb07 32820db29a49 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (7m) 93s ago 19m 55.0M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 72e8f30ba3f4 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi149 running (6m) 94s ago 18m 43.0M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 896e9f535de0 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (7m) 93s ago 18m 47.2M 2048M 19.0.0-3198-g56e81df3 82086b19bb07 748d9311f48f 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (8m) 93s ago 14m 17.5M - 1.3.1 1dbe0e931976 45fdbfb25545 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi149 *:9100 running (8m) 94s ago 14m 16.8M - 1.3.1 1dbe0e931976 2d32c60dbd33 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (6m) 93s ago 17m 55.9M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 f6954c3f3c2a 2024-04-23T03:39:39.423 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (5m) 93s ago 17m 57.5M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 876b7dc7ea6d 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (5m) 93s ago 16m 54.3M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 9c59ba461d53 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (4m) 93s ago 16m 65.0M 2737M 19.0.0-3198-g56e81df3 82086b19bb07 e39ff5d76409 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi149 running (4m) 94s ago 16m 57.9M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 0a604e140ced 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi149 running (3m) 94s ago 15m 51.0M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 f88b3de0b1f7 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi149 running (3m) 94s ago 15m 62.2M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 ec30c8d5356b 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi149 running (2m) 94s ago 15m 61.8M 1454M 19.0.0-3198-g56e81df3 82086b19bb07 7a072e4b062d 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi149 *:9095 running (7m) 94s ago 14m 76.3M - 2.33.4 514e6a882f6e 86ffd466d290 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.bipufo smithi007 *:8000 running (2m) 93s ago 12m 76.6M - 19.0.0-3198-g56e81df3 82086b19bb07 2460b2288b6e 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi149.axrzpl smithi149 *:8000 running (2m) 94s ago 12m 77.7M - 19.0.0-3198-g56e81df3 82086b19bb07 6d2a2f0efe30 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.mxntqi smithi007 *:80 running (2m) 93s ago 12m 76.4M - 19.0.0-3198-g56e81df3 82086b19bb07 fcea1228ccfd 2024-04-23T03:39:39.424 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi149.rtdeuy smithi149 *:80 running (2m) 94s ago 12m 75.5M - 19.0.0-3198-g56e81df3 82086b19bb07 ebb99282cbed 2024-04-23T03:39:40.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph versions' 2024-04-23T03:39:40.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:40 smithi007 ceph-mon[64248]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:40.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:40 smithi007 ceph-mon[61772]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:40.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:40 smithi149 ceph-mon[54848]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:41.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:41 smithi007 ceph-mon[61772]: from='client.44781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:41.583 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:41 smithi007 ceph-mon[64248]: from='client.44781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:41.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:41 smithi149 ceph-mon[54848]: from='client.44781 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:42.394 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:39:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-23T03:39:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 3 2024-04-23T03:39:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:39:42.394 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 2 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 8 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 4 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)": 17 2024-04-23T03:39:42.395 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-23T03:39:42.396 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:39:42.573 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:42 smithi149 ceph-mon[54848]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:42.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:42 smithi007 ceph-mon[64248]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:42.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:42 smithi007 ceph-mon[61772]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:43.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch upgrade status' 2024-04-23T03:39:43.332 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:43 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/686562277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:43.333 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:43 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/686562277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:43.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:43 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/686562277' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:44.503 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:44 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:44.503 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:44 smithi007 ceph-mon[64248]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:44.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:44 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:44.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:44 smithi007 ceph-mon[61772]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:44.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:44 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:44.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:44 smithi149 ceph-mon[54848]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7", 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "which": "Upgrading all daemon types on all hosts", 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [ 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "mon", 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr", 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw", 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: "osd" 2024-04-23T03:39:45.093 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-23T03:39:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "17/23 daemons upgraded", 2024-04-23T03:39:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed.", 2024-04-23T03:39:45.094 INFO:teuthology.orchestra.run.smithi007.stdout: "is_paused": true 2024-04-23T03:39:45.094 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:39:45.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph health detail' 2024-04-23T03:39:46.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-mon[64248]: from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:46.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-mon[64248]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:46.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:46.351 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-mon[61772]: from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:46.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-mon[61772]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:46.352 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:46.352 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:39:46 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:39:46] "GET /metrics HTTP/1.1" 200 38686 "" "Prometheus/2.33.4" 2024-04-23T03:39:46.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:46 smithi149 ceph-mon[54848]: from='client.54721 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:46 smithi149 ceph-mon[54848]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:46.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:46 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrade daemon: iscsi.foo.smithi007.xwmqjz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout:Creating ceph-iscsi config... 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout:Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:39:47.833 INFO:teuthology.orchestra.run.smithi007.stdout:Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3d05926e\x2d0120\x2d11ef\x2dbc93\x2dc7b262605968.slice/ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 1 from systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout:systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout:Traceback (most recent call last): 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout: return _run_code(code, main_globals, None, 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout: exec(code, run_globals) 2024-04-23T03:39:47.834 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1087, in deploy_daemon 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1223, in deploy_daemon_units 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout:RuntimeError: Failed command: systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:39:47.835 INFO:teuthology.orchestra.run.smithi007.stdout:See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:39:48.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:48 smithi007 ceph-mon[64248]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:48.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:48 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2008319111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:39:48.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:48 smithi007 ceph-mon[61772]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:48.539 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:48 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2008319111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:39:48.565 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-23T03:39:48.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:48 smithi149 ceph-mon[54848]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:48.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:48 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2008319111' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-23T03:39:50.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:50 smithi007 ceph-mon[64248]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:50.583 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:50 smithi007 ceph-mon[61772]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:50.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:50 smithi149 ceph-mon[54848]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:50.713 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-23T03:39:51.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-23T03:39:51.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:51 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/1216492640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:51.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:51 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/1216492640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:51.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:51 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/1216492640' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:52.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:52 smithi007 ceph-mon[61772]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:52.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:52 smithi007 ceph-mon[64248]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:52.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:52 smithi149 ceph-mon[54848]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:53.281 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 19.0.0-3198-g56e81df3 (56e81df3aeb98b717efe2ab0537a8c60249f95f7) squid (dev)" 2024-04-23T03:39:53.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:53 smithi007 ceph-mon[64248]: from='client.? 172.21.15.7:0/2838548197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:53.567 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:53 smithi007 ceph-mon[61772]: from='client.? 172.21.15.7:0/2838548197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:53.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:53 smithi149 ceph-mon[54848]: from='client.? 172.21.15.7:0/2838548197' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-23T03:39:53.876 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -e sha1=56e81df3aeb98b717efe2ab0537a8c60249f95f7 -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-04-23T03:39:54.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:54 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:54.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:54 smithi007 ceph-mon[61772]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:54.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:54 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:54.513 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:54 smithi007 ceph-mon[64248]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:54.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:54 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:39:54.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:54 smithi149 ceph-mon[54848]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:39:55.812 INFO:teuthology.orchestra.run.smithi007.stdout:osd 8 110s ago - 2024-04-23T03:39:56.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:56 smithi007 ceph-mon[64248]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:56.515 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:39:56 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:39:56] "GET /metrics HTTP/1.1" 200 38686 "" "Prometheus/2.33.4" 2024-04-23T03:39:56.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:56 smithi007 ceph-mon[61772]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:56.541 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-23T03:39:56.551 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-23T03:39:56.551 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-04-23T03:39:56.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:56 smithi149 ceph-mon[54848]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:57.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:57 smithi007 ceph-mon[64248]: from='client.44817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:57.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:57 smithi007 ceph-mon[61772]: from='client.44817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:57.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:57 smithi149 ceph-mon[54848]: from='client.44817 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:39:58.582 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:39:58 smithi007 ceph-mon[61772]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:58.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:39:58 smithi007 ceph-mon[64248]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:58.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:39:58 smithi149 ceph-mon[54848]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:39:59.866 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-23T03:39:59.866 INFO:teuthology.orchestra.run.smithi007.stdout: "image": "quay.io/ceph/ceph", 2024-04-23T03:39:59.866 INFO:teuthology.orchestra.run.smithi007.stdout: "registry": "quay.io", 2024-04-23T03:39:59.866 INFO:teuthology.orchestra.run.smithi007.stdout: "bare_image": "ceph/ceph", 2024-04-23T03:39:59.866 INFO:teuthology.orchestra.run.smithi007.stdout: "versions": [] 2024-04-23T03:39:59.867 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Upgrade daemon: iscsi.foo.smithi007.xwmqjz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Creating ceph-iscsi config... 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:40:00.497 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3d05926e\x2d0120\x2d11ef\x2dbc93\x2dc7b262605968.slice/ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Non-zero exit code 1 from systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: Traceback (most recent call last): 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: return _run_code(code, main_globals, None, 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: exec(code, run_globals) 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:40:00.498 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1087, in deploy_daemon 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Upgrade daemon: iscsi.foo.smithi007.xwmqjz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Creating ceph-iscsi config... 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3d05926e\x2d0120\x2d11ef\x2dbc93\x2dc7b262605968.slice/ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service 2024-04-23T03:40:00.499 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Non-zero exit code 1 from systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: Traceback (most recent call last): 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: return _run_code(code, main_globals, None, 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: exec(code, run_globals) 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:40:00.500 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1087, in deploy_daemon 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1223, in deploy_daemon_units 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: RuntimeError: Failed command: systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[64248]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1223, in deploy_daemon_units 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: RuntimeError: Failed command: systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:40:00.501 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:00 smithi007 ceph-mon[61772]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:40:00.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: from='client.54751 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: daemon iscsi.foo.smithi007.xwmqjz on smithi007 is in unknown state 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: [WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi007.xwmqjz on host smithi007 failed. 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Upgrade daemon: iscsi.foo.smithi007.xwmqjz: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi007.xwmqjz ... 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Creating ceph-iscsi config... 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/iscsi-gateway.cfg 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Write file: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/iscsi.foo.smithi007.xwmqjz/tcmu-runner-entrypoint.sh 2024-04-23T03:40:00.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d3d05926e\x2d0120\x2d11ef\x2dbc93\x2dc7b262605968.slice/ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Non-zero exit code 1 from systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: systemctl: stderr Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: systemctl: stderr See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: Traceback (most recent call last): 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: return _run_code(code, main_globals, None, 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: exec(code, run_globals) 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5618, in 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 5606, in main 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3096, in command_deploy_from 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3131, in _common_deploy 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 3151, in _deploy_daemon_container 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1087, in deploy_daemon 2024-04-23T03:40:00.673 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/__main__.py", line 1223, in deploy_daemon_units 2024-04-23T03:40:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: File "/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/cephadm.e31ca4716d00da92240374b06df95e06dd890b951fabc12934151562def20805/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-04-23T03:40:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: RuntimeError: Failed command: systemctl start ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz: Job for ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service failed because the control process exited with error code. 2024-04-23T03:40:00.674 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:00 smithi149 ceph-mon[54848]: See "systemctl status ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" and "journalctl -xeu ceph-3d05926e-0120-11ef-bc93-c7b262605968@iscsi.foo.smithi007.xwmqjz.service" for details. 2024-04-23T03:40:01.554 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:01 smithi007 ceph-mon[64248]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:40:01.554 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:01 smithi007 ceph-mon[61772]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:40:01.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:01 smithi149 ceph-mon[54848]: from='mgr.25354 172.21.15.7:0/3857829226' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-04-23T03:40:02.472 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:02 smithi007 ceph-mon[64248]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:40:02.472 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:02 smithi007 ceph-mon[61772]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:40:02.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:02 smithi149 ceph-mon[54848]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:40:03.135 INFO:teuthology.orchestra.run.smithi007.stdout: "16.2.0", 2024-04-23T03:40:03.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3d05926e-0120-11ef-bc93-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-04-23T03:40:04.369 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:04 smithi007 ceph-mon[64248]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:40:04.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:04 smithi007 ceph-mon[64248]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:04.370 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:04 smithi007 ceph-mon[64248]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:40:04.370 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:04 smithi007 ceph-mon[61772]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:40:04.370 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:04 smithi007 ceph-mon[61772]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:04.370 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:04 smithi007 ceph-mon[61772]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:40:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:04 smithi149 ceph-mon[54848]: from='client.54634 -' entity='client.iscsi.foo.smithi007.xwmqjz' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-23T03:40:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:04 smithi149 ceph-mon[54848]: from='client.54757 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:04.672 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:04 smithi149 ceph-mon[54848]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-23T03:40:06.418 INFO:teuthology.orchestra.run.smithi007.stdout: "v16.2.2", 2024-04-23T03:40:06.418 INFO:teuthology.orchestra.run.smithi007.stdout: "v16.2.2-20210505", 2024-04-23T03:40:06.419 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:06 smithi007 ceph-mon[61772]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:40:06.419 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:06 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: ::ffff:172.21.15.149 - - [23/Apr/2024:03:40:06] "GET /metrics HTTP/1.1" 200 38689 "" "Prometheus/2.33.4" 2024-04-23T03:40:06.419 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:06 smithi007 ceph-mon[64248]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:40:06.671 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:06 smithi149 ceph-mon[54848]: pgmap v289: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-23T03:40:07.093 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-23T03:40:07.104 INFO:tasks.cephadm:Teardown begin 2024-04-23T03:40:07.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:40:07.132 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:40:07.166 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-23T03:40:07.166 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T03:40:07.188 DEBUG:teuthology.orchestra.run.smithi149:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-23T03:40:07.224 INFO:tasks.cephadm:Stopping all daemons... 2024-04-23T03:40:07.224 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-23T03:40:07.224 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a 2024-04-23T03:40:07.332 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:07 smithi007 systemd[1]: Stopping Ceph mon.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:07.421 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:07 smithi149 ceph-mon[54848]: from='client.54763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:07.672 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:07 smithi007 ceph-mon[64248]: from='client.54763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:07.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:07 smithi007 ceph-mon[61772]: from='client.54763 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-23T03:40:07.672 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:07 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a[61768]: 2024-04-23T03:40:07.563+0000 7fd4aebc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:07.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:07 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a[61768]: 2024-04-23T03:40:07.563+0000 7fd4aebc7700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-04-23T03:40:08.055 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:07 smithi007 podman[88076]: 2024-04-23 03:40:07.757380286 +0000 UTC m=+0.310635927 container died 72e8f30ba3f4d10d4826f7d2a11c2f3f9bad187825c9a0b845f0d61992969d1d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240416) 2024-04-23T03:40:08.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:08 smithi007 podman[88076]: 2024-04-23 03:40:08.146094768 +0000 UTC m=+0.699350417 container cleanup 72e8f30ba3f4d10d4826f7d2a11c2f3f9bad187825c9a0b845f0d61992969d1d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df) 2024-04-23T03:40:08.335 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 23 03:40:08 smithi007 bash[88076]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-a 2024-04-23T03:40:08.576 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.a.service' 2024-04-23T03:40:08.626 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:08.626 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-23T03:40:08.626 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-23T03:40:08.626 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.c 2024-04-23T03:40:08.832 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:08 smithi007 systemd[1]: Stopping Ceph mon.c for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:09.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c[64241]: 2024-04-23T03:40:08.933+0000 7f77b9230700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:09.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c[64241]: 2024-04-23T03:40:08.933+0000 7f77b9230700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-04-23T03:40:09.302 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:09 smithi007 podman[88228]: 2024-04-23 03:40:09.036478482 +0000 UTC m=+0.225728869 container died 748d9311f48fa93189f5afeaa1ddfe10c921ac2a86593df4f05ef6a46dcd973f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-23T03:40:09.302 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:40:08] ENGINE Bus STOPPING 2024-04-23T03:40:09.302 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:40:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-04-23T03:40:09.302 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:40:08] ENGINE Bus STOPPED 2024-04-23T03:40:09.303 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:08 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:40:08] ENGINE Bus STARTING 2024-04-23T03:40:09.303 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: CherryPy Checker: 2024-04-23T03:40:09.303 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: The Application mounted at '' has an empty config. 2024-04-23T03:40:09.303 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:40:09.303 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:40:09] ENGINE Serving on http://:::9283 2024-04-23T03:40:09.303 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:09 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: [23/Apr/2024:03:40:09] ENGINE Bus STARTED 2024-04-23T03:40:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:09 smithi007 podman[88228]: 2024-04-23 03:40:09.426108521 +0000 UTC m=+0.615358910 container cleanup 748d9311f48fa93189f5afeaa1ddfe10c921ac2a86593df4f05ef6a46dcd973f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0) 2024-04-23T03:40:09.582 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 23 03:40:09 smithi007 bash[88228]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-c 2024-04-23T03:40:09.847 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.c.service' 2024-04-23T03:40:09.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:09.883 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-23T03:40:09.883 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-23T03:40:09.883 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.b 2024-04-23T03:40:10.171 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:09 smithi149 systemd[1]: Stopping Ceph mon.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:10.633 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b[54844]: 2024-04-23T03:40:10.197+0000 7fa1620fb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:10.633 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b[54844]: 2024-04-23T03:40:10.197+0000 7fa1620fb700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-04-23T03:40:10.633 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:10 smithi149 podman[73075]: 2024-04-23 03:40:10.331942942 +0000 UTC m=+0.263918900 container died 896e9f535de0c41ac76717ea6f8829cb81feba99e0ed3613e8d44a651b5e357e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, CEPH_POINT_RELEASE=) 2024-04-23T03:40:10.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:10 smithi149 podman[73075]: 2024-04-23 03:40:10.774121043 +0000 UTC m=+0.706097003 container cleanup 896e9f535de0c41ac76717ea6f8829cb81feba99e0ed3613e8d44a651b5e357e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:10.922 INFO:journalctl@ceph.mon.b.smithi149.stdout:Apr 23 03:40:10 smithi149 bash[73075]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mon-b 2024-04-23T03:40:11.196 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mon.b.service' 2024-04-23T03:40:11.237 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:11.237 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-23T03:40:11.237 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-23T03:40:11.237 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y 2024-04-23T03:40:11.539 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:11 smithi007 systemd[1]: Stopping Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:11.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:40:11.538+0000 7f1cae84c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:11.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:11 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y[57358]: 2024-04-23T03:40:11.538+0000 7f1cae84c700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-23T03:40:11.815 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:11 smithi007 podman[88359]: 2024-04-23 03:40:11.57100447 +0000 UTC m=+0.139816755 container died 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, RELEASE=main-56e81df, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-04-23T03:40:12.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:11 smithi007 podman[88359]: 2024-04-23 03:40:11.897176141 +0000 UTC m=+0.465988419 container cleanup 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-23T03:40:12.082 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:11 smithi007 bash[88359]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y 2024-04-23T03:40:12.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-23T03:40:12.360 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:12 smithi007 podman[88373]: 2024-04-23 03:40:12.098940719 +0000 UTC m=+0.524690367 container remove 32820db29a49095fb7e5f3b3ab3754bef56401b8c54eae01e0432b7115fbca15 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-y, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-04-23T03:40:12.361 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:12 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service: Deactivated successfully. 2024-04-23T03:40:12.361 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:12 smithi007 systemd[1]: Stopped Ceph mgr.y for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:40:12.361 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 23 03:40:12 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.y.service: Consumed 41.816s CPU time. 2024-04-23T03:40:12.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:12.375 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-23T03:40:12.375 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-23T03:40:12.375 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x 2024-04-23T03:40:12.671 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:40:12 smithi149 systemd[1]: Stopping Ceph mgr.x for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:13.046 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:40:12 smithi149 podman[73194]: 2024-04-23 03:40:12.750895892 +0000 UTC m=+0.191267291 container died d525caf93187836cd5751d67498e5a1410f96113d3c5ae0bae251fdd6ff2fb61 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-23T03:40:13.421 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:40:13 smithi149 podman[73194]: 2024-04-23 03:40:13.203773661 +0000 UTC m=+0.644145058 container cleanup d525caf93187836cd5751d67498e5a1410f96113d3c5ae0bae251fdd6ff2fb61 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:40:13.422 INFO:journalctl@ceph.mgr.x.smithi149.stdout:Apr 23 03:40:13 smithi149 bash[73194]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-mgr-x 2024-04-23T03:40:13.583 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-23T03:40:13.620 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:13.620 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-23T03:40:13.620 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-23T03:40:13.621 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.0 2024-04-23T03:40:13.927 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:13 smithi007 systemd[1]: Stopping Ceph osd.0 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:14.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:40:13.926+0000 7f6855e5d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:14.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:40:13.926+0000 7f6855e5d700 -1 osd.0 152 *** Got signal Terminated *** 2024-04-23T03:40:14.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:13 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0[67564]: 2024-04-23T03:40:13.926+0000 7f6855e5d700 -1 osd.0 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:40:19.582 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:19 smithi007 podman[88479]: 2024-04-23 03:40:19.330927559 +0000 UTC m=+5.512376712 container died f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-23T03:40:19.910 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:19 smithi007 podman[88479]: 2024-04-23 03:40:19.659384498 +0000 UTC m=+5.840833659 container cleanup f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:19.910 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:19 smithi007 bash[88479]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0 2024-04-23T03:40:20.332 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:19 smithi007 podman[88496]: 2024-04-23 03:40:19.910522125 +0000 UTC m=+0.578177724 container remove f6954c3f3c2a4812e386d47ed6b0d3a6f7789611921d401758e485017c70bb0f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-23T03:40:20.605 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:20 smithi007 podman[88588]: 2024-04-23 03:40:20.239546978 +0000 UTC m=+0.017880830 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:40:20.605 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:20 smithi007 podman[88588]: 2024-04-23 03:40:20.344615458 +0000 UTC m=+0.122949285 container create 776d2ba5e7a83180340bf28aae397b7439e63705e0c87e08284807be3a5e4f88 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:20.605 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:20 smithi007 podman[88588]: 2024-04-23 03:40:20.547691023 +0000 UTC m=+0.326024865 container init 776d2ba5e7a83180340bf28aae397b7439e63705e0c87e08284807be3a5e4f88 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:40:20.605 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:20 smithi007 podman[88588]: 2024-04-23 03:40:20.551288003 +0000 UTC m=+0.329621832 container start 776d2ba5e7a83180340bf28aae397b7439e63705e0c87e08284807be3a5e4f88 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-23T03:40:20.606 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:20 smithi007 podman[88588]: 2024-04-23 03:40:20.605413426 +0000 UTC m=+0.383747266 container attach 776d2ba5e7a83180340bf28aae397b7439e63705e0c87e08284807be3a5e4f88 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240416) 2024-04-23T03:40:21.159 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 23 03:40:20 smithi007 podman[88588]: 2024-04-23 03:40:20.873772961 +0000 UTC m=+0.652106796 container died 776d2ba5e7a83180340bf28aae397b7439e63705e0c87e08284807be3a5e4f88 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-23T03:40:21.454 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.0.service' 2024-04-23T03:40:21.499 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:21.499 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-23T03:40:21.499 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-23T03:40:21.499 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.1 2024-04-23T03:40:21.726 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:21 smithi007 systemd[1]: Stopping Ceph osd.1 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:22.082 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:40:21.869+0000 7fa6a1354700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:22.082 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:40:21.869+0000 7fa6a1354700 -1 osd.1 152 *** Got signal Terminated *** 2024-04-23T03:40:22.083 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1[70105]: 2024-04-23T03:40:21.869+0000 7fa6a1354700 -1 osd.1 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:40:24.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:40:24.583 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:24 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:24.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:40:27.704 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:27 smithi007 podman[88694]: 2024-04-23 03:40:27.36502485 +0000 UTC m=+5.606981527 container died 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, ceph=True) 2024-04-23T03:40:27.705 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:27 smithi007 podman[88694]: 2024-04-23 03:40:27.703813598 +0000 UTC m=+5.945770268 container cleanup 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-23T03:40:28.082 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:27 smithi007 bash[88694]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1 2024-04-23T03:40:28.082 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:27 smithi007 podman[88723]: 2024-04-23 03:40:27.90547038 +0000 UTC m=+0.538642238 container remove 876b7dc7ea6d2b5dfc68044fe194198d229077acb5a2bd3e2a212a0851b80556 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, io.buildah.version=1.33.5, RELEASE=main-56e81df, ceph=True) 2024-04-23T03:40:28.596 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:28 smithi007 podman[88815]: 2024-04-23 03:40:28.244764416 +0000 UTC m=+0.018560577 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:40:28.596 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:28 smithi007 podman[88815]: 2024-04-23 03:40:28.422980114 +0000 UTC m=+0.196776275 container create b374f3b6411aa259ba3c37f856b96e9da4d7f64b8fee1dfca7f025fbbe444f84 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-04-23T03:40:29.005 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:28 smithi007 podman[88815]: 2024-04-23 03:40:28.692093018 +0000 UTC m=+0.465889181 container init b374f3b6411aa259ba3c37f856b96e9da4d7f64b8fee1dfca7f025fbbe444f84 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:40:29.005 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:28 smithi007 podman[88815]: 2024-04-23 03:40:28.695355063 +0000 UTC m=+0.469151226 container start b374f3b6411aa259ba3c37f856b96e9da4d7f64b8fee1dfca7f025fbbe444f84 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240416, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T03:40:29.005 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:28 smithi007 podman[88815]: 2024-04-23 03:40:28.767014294 +0000 UTC m=+0.540810459 container attach b374f3b6411aa259ba3c37f856b96e9da4d7f64b8fee1dfca7f025fbbe444f84 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:40:29.303 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 23 03:40:29 smithi007 podman[88815]: 2024-04-23 03:40:29.005005599 +0000 UTC m=+0.778801772 container died b374f3b6411aa259ba3c37f856b96e9da4d7f64b8fee1dfca7f025fbbe444f84 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-23T03:40:29.579 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.1.service' 2024-04-23T03:40:29.621 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:29.622 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-23T03:40:29.622 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-23T03:40:29.622 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.2 2024-04-23T03:40:29.832 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:29 smithi007 systemd[1]: Stopping Ceph osd.2 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:30.332 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:29 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:40:29.947+0000 7fa558704700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:30.332 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:29 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:40:29.947+0000 7fa558704700 -1 osd.2 152 *** Got signal Terminated *** 2024-04-23T03:40:30.332 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:29 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2[72497]: 2024-04-23T03:40:29.947+0000 7fa558704700 -1 osd.2 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:40:34.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=error ts=2024-04-23T03:40:34.240Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:40:34.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:34.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:40:34.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:34 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:34.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:40:35.707 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:35 smithi007 podman[88921]: 2024-04-23 03:40:35.44584436 +0000 UTC m=+5.592630713 container died 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416) 2024-04-23T03:40:36.082 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:35 smithi007 podman[88921]: 2024-04-23 03:40:35.806346286 +0000 UTC m=+5.953132646 container cleanup 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-56e81df, CEPH_POINT_RELEASE=) 2024-04-23T03:40:36.082 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:35 smithi007 bash[88921]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2 2024-04-23T03:40:36.488 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:36 smithi007 podman[88938]: 2024-04-23 03:40:36.132879464 +0000 UTC m=+0.690231692 container remove 9c59ba461d53cda437b1ebd499493dc546b66739d134155c46e9b89470e51549 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-04-23T03:40:36.739 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:36 smithi007 podman[89029]: 2024-04-23 03:40:36.391443579 +0000 UTC m=+0.018304281 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:40:36.739 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:36 smithi007 podman[89029]: 2024-04-23 03:40:36.520115074 +0000 UTC m=+0.146975792 container create 175870c0449fe1b912b41146ed0228544c93150bafc98d95b1985dbbe9f8a148 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-23T03:40:36.739 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:36 smithi007 podman[89029]: 2024-04-23 03:40:36.736357045 +0000 UTC m=+0.363217718 container init 175870c0449fe1b912b41146ed0228544c93150bafc98d95b1985dbbe9f8a148 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240416, RELEASE=main-56e81df, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-23T03:40:36.740 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:36 smithi007 podman[89029]: 2024-04-23 03:40:36.739499819 +0000 UTC m=+0.366360493 container start 175870c0449fe1b912b41146ed0228544c93150bafc98d95b1985dbbe9f8a148 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:37.049 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:36 smithi007 podman[89029]: 2024-04-23 03:40:36.836370505 +0000 UTC m=+0.463231194 container attach 175870c0449fe1b912b41146ed0228544c93150bafc98d95b1985dbbe9f8a148 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-23T03:40:37.301 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 23 03:40:37 smithi007 podman[89029]: 2024-04-23 03:40:37.049655524 +0000 UTC m=+0.676516209 container died 175870c0449fe1b912b41146ed0228544c93150bafc98d95b1985dbbe9f8a148 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-23T03:40:37.729 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.2.service' 2024-04-23T03:40:37.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:37.768 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-23T03:40:37.768 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-23T03:40:37.768 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.3 2024-04-23T03:40:38.081 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:37 smithi007 systemd[1]: Stopping Ceph osd.3 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:38.582 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:40:38.125+0000 7f5e491a3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:38.582 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:40:38.125+0000 7f5e491a3700 -1 osd.3 152 *** Got signal Terminated *** 2024-04-23T03:40:38.582 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:38 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:40:38.125+0000 7f5e491a3700 -1 osd.3 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:40:41.000 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:40 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:40.725+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:41.421 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:40 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:40.998+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:42.000 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:41 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:41.684+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:42.417 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:42 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:41.999+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:43.040 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:42 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:42.700+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:43.082 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:42 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3[74964]: 2024-04-23T03:40:42.675+0000 7f5e4178f700 -1 osd.3 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:18.910377+0000 front 2024-04-23T03:40:18.910208+0000 (oldest deadline 2024-04-23T03:40:41.810436+0000) 2024-04-23T03:40:43.422 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:43.039+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:43.727 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:43 smithi007 podman[89145]: 2024-04-23 03:40:43.474742455 +0000 UTC m=+5.482156514 container died e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:40:43.906 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:43.675+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:43.906 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:43.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:44.010 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:43 smithi007 podman[89145]: 2024-04-23 03:40:43.817164428 +0000 UTC m=+5.824578479 container cleanup e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, ceph=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:40:44.010 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:43 smithi007 bash[89145]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3 2024-04-23T03:40:44.171 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:44.044+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:44.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:43 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:43.905+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:44.332 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89163]: 2024-04-23 03:40:44.010110825 +0000 UTC m=+0.530803973 container remove e39ff5d7640913a19e6fd2f49e6869e7284d95617e90799c809ace222c3a1a1c (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, ceph=True, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:44.332 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=error ts=2024-04-23T03:40:44.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:40:44.332 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:44.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:40:44.332 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:44 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:44.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:40:44.721 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89252]: 2024-04-23 03:40:44.41907623 +0000 UTC m=+0.110721958 container create f6cbe94b80983f4db00037e1ff4fd6c18cbf4982173a615fd8ca158ba0a08307 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416) 2024-04-23T03:40:44.722 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89252]: 2024-04-23 03:40:44.327066246 +0000 UTC m=+0.018711986 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:40:44.722 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89252]: 2024-04-23 03:40:44.638387293 +0000 UTC m=+0.330033022 container init f6cbe94b80983f4db00037e1ff4fd6c18cbf4982173a615fd8ca158ba0a08307 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-23T03:40:44.722 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89252]: 2024-04-23 03:40:44.641630504 +0000 UTC m=+0.333276232 container start f6cbe94b80983f4db00037e1ff4fd6c18cbf4982173a615fd8ca158ba0a08307 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, org.label-schema.build-date=20240416, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-04-23T03:40:44.722 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89252]: 2024-04-23 03:40:44.721646845 +0000 UTC m=+0.413292589 container attach f6cbe94b80983f4db00037e1ff4fd6c18cbf4982173a615fd8ca158ba0a08307 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240416, ceph=True) 2024-04-23T03:40:44.914 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:44.652+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:44.914 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:44.620+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:45.082 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:44 smithi007 podman[89252]: 2024-04-23 03:40:44.963802739 +0000 UTC m=+0.655448460 container died f6cbe94b80983f4db00037e1ff4fd6c18cbf4982173a615fd8ca158ba0a08307 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-23T03:40:45.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:44 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:44.913+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:45.172 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:45.030+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:45.516 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 23 03:40:45 smithi007 podman[89278]: 2024-04-23 03:40:45.491543148 +0000 UTC m=+0.522702572 container remove f6cbe94b80983f4db00037e1ff4fd6c18cbf4982173a615fd8ca158ba0a08307 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:40:45.522 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.3.service' 2024-04-23T03:40:45.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:45.567 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-23T03:40:45.567 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-23T03:40:45.567 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.4 2024-04-23T03:40:45.844 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:45 smithi149 systemd[1]: Stopping Ceph osd.4 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:45.845 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:45.623+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:45.845 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:40:45.839Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-04-23T03:40:45.846 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:45.668+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:46.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:45.941+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:46.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:40:45.844Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-04-23T03:40:46.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:40:45.844Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-04-23T03:40:46.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:40:45.844Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-04-23T03:40:46.172 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:40:45.844Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.7:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.7:8765: connect: connection refused" 2024-04-23T03:40:46.173 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:45.871+0000 7f7935a88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:46.173 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:45.871+0000 7f7935a88700 -1 osd.4 152 *** Got signal Terminated *** 2024-04-23T03:40:46.173 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:45 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:45.871+0000 7f7935a88700 -1 osd.4 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:40:46.173 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:46 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:46.013+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:46.921 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:46 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:46.623+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:46.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:46 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:46.630+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:47.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:46 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:46.990+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:47.422 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:47.059+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:47.921 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:47.625+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:47.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:47 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:47.585+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:48.421 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:48.037+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:48.422 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:48.045+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:48.921 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:48.588+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:48.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:48.564+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:48.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:48.564+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:49.421 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:48 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:48.990+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:49.421 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:49 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:49.093+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:49.921 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:49 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:49.551+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:49.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:49 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:49.537+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:49.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:49 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:49.537+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:50.422 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4[58023]: 2024-04-23T03:40:50.081+0000 7f793207c700 -1 osd.4 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:16.876209+0000 front 2024-04-23T03:40:16.876120+0000 (oldest deadline 2024-04-23T03:40:40.976118+0000) 2024-04-23T03:40:50.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:49 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:49.982+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:50.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:49 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:49.982+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:50.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:50.570+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:50.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:50.570+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:50.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:50.592+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:50.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:50.592+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:51.262 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:50.947+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:51.262 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:50 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:50.947+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:51.514 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:51 smithi149 podman[73343]: 2024-04-23 03:40:51.26191252 +0000 UTC m=+5.498844741 container died 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:51.905 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:51.557+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:51.905 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:51.557+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:51.905 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:51.601+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:51.905 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:51.601+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:52.171 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:51 smithi149 podman[73343]: 2024-04-23 03:40:51.905200693 +0000 UTC m=+6.142132913 container cleanup 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:40:52.172 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:51 smithi149 bash[73343]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4 2024-04-23T03:40:52.172 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73360]: 2024-04-23 03:40:52.056623695 +0000 UTC m=+0.791582050 container remove 0a604e140ced34c635ea7ba70433c6f60bcdbecbeb4e8aa13eea794f3447bf60 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-23T03:40:52.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:51.979+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:52.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:51 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:51.979+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:52.718 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:52.523+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:52.718 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:52.523+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:52.607+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:52.607+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73451]: 2024-04-23 03:40:52.457398637 +0000 UTC m=+0.087037009 container create 70033c520918491936ba656607d4be8b13c8b8b31b8809e817a018bb198c951f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-56e81df, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73451]: 2024-04-23 03:40:52.388801253 +0000 UTC m=+0.018439594 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73451]: 2024-04-23 03:40:52.618917985 +0000 UTC m=+0.248556311 container init 70033c520918491936ba656607d4be8b13c8b8b31b8809e817a018bb198c951f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-56e81df) 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73451]: 2024-04-23 03:40:52.622278815 +0000 UTC m=+0.251917151 container start 70033c520918491936ba656607d4be8b13c8b8b31b8809e817a018bb198c951f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux ) 2024-04-23T03:40:52.719 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73451]: 2024-04-23 03:40:52.718222271 +0000 UTC m=+0.347860611 container attach 70033c520918491936ba656607d4be8b13c8b8b31b8809e817a018bb198c951f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df) 2024-04-23T03:40:53.172 INFO:journalctl@ceph.osd.4.smithi149.stdout:Apr 23 03:40:52 smithi149 podman[73451]: 2024-04-23 03:40:52.921526459 +0000 UTC m=+0.551164779 container died 70033c520918491936ba656607d4be8b13c8b8b31b8809e817a018bb198c951f (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-23T03:40:53.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:52.943+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:53.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:52 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:52.943+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:53.778 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.4.service' 2024-04-23T03:40:53.815 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:53.554+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:53.815 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:53.554+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:53.815 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:53.575+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:53.815 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:53.575+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:53.817 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:40:53.817 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-23T03:40:53.817 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-23T03:40:53.818 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.5 2024-04-23T03:40:54.142 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:53 smithi149 systemd[1]: Stopping Ceph osd.5 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:40:54.142 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:53.964+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:54.142 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:53 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:53.964+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:54.408 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:54.141+0000 7fdda1a89700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:40:54.408 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:54.141+0000 7fdda1a89700 -1 osd.5 152 *** Got signal Terminated *** 2024-04-23T03:40:54.408 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:54.141+0000 7fdda1a89700 -1 osd.5 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:40:54.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:54 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=error ts=2024-04-23T03:40:54.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:40:54.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:54 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:54.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:40:54.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:40:54 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:40:54.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:40:54.672 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:54.609+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:54.672 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:54.609+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:54.672 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:54.523+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:54.672 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:54.523+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:55.421 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:54.936+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:55.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:54 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:54.936+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:55.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:55.482+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:55.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:55.482+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:55.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:55.641+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:55.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:55.641+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:56.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:55.980+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:56.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:55 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:55.980+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:56.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:56.492+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:56.923 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:56.492+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:56.923 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:56.492+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.771809+0000 front 2024-04-23T03:40:32.771743+0000 (oldest deadline 2024-04-23T03:40:56.271744+0000) 2024-04-23T03:40:56.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:56.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:56.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:56.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:56.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:56.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:40:57.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:56.938+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:57.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:56 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:56.938+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:57.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:57.466+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:57.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:57.466+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:57.922 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:57.466+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.771809+0000 front 2024-04-23T03:40:32.771743+0000 (oldest deadline 2024-04-23T03:40:56.271744+0000) 2024-04-23T03:40:57.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:57.645+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:57.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:57.645+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:57.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:57.645+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:40:58.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:57.924+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:58.423 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:57 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:57.924+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:58.882 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:58.446+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:15.969379+0000 front 2024-04-23T03:40:15.969459+0000 (oldest deadline 2024-04-23T03:40:40.069165+0000) 2024-04-23T03:40:58.883 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:58.446+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:25.770991+0000 front 2024-04-23T03:40:25.770926+0000 (oldest deadline 2024-04-23T03:40:49.870753+0000) 2024-04-23T03:40:58.883 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5[60363]: 2024-04-23T03:40:58.446+0000 7fdd9e07d700 -1 osd.5 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.771809+0000 front 2024-04-23T03:40:32.771743+0000 (oldest deadline 2024-04-23T03:40:56.271744+0000) 2024-04-23T03:40:58.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:58.638+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:58.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:58.638+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:58.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:58.638+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:40:59.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:58.880+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:40:59.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:58.880+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:40:59.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:58.880+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:40:59.882 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:40:59 smithi149 podman[73567]: 2024-04-23 03:40:59.613259978 +0000 UTC m=+5.556822472 container died f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-56e81df, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-23T03:40:59.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:59.675+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:40:59.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:59.675+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:40:59.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:40:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:40:59.675+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:00.172 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 podman[73567]: 2024-04-23 03:41:00.033507136 +0000 UTC m=+5.977069637 container cleanup f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T03:41:00.172 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 bash[73567]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5 2024-04-23T03:41:00.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:59.880+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:00.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:59.880+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:00.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:40:59 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:40:59.880+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:00.672 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 podman[73585]: 2024-04-23 03:41:00.268155156 +0000 UTC m=+0.648766940 container remove f88b3de0b1f79d0d9796a5d9f36675a0be03a03e8a302207d68e458041d847a4 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-56e81df, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-23T03:41:00.954 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:00.876+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:00.954 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:00.876+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:00.955 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:00.876+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:00.955 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:00.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:00.955 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:00.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:00.955 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:00 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:00.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:00.955 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 podman[73680]: 2024-04-23 03:41:00.602352791 +0000 UTC m=+0.018778476 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:41:00.956 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 podman[73680]: 2024-04-23 03:41:00.718662802 +0000 UTC m=+0.135088473 container create 79ec9ad95f45f6c5692b956af68f4c6e9e4a60b5705279577ba13adc22ce6202 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.schema-version=1.0) 2024-04-23T03:41:00.956 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 podman[73680]: 2024-04-23 03:41:00.954293877 +0000 UTC m=+0.370719549 container init 79ec9ad95f45f6c5692b956af68f4c6e9e4a60b5705279577ba13adc22ce6202 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-56e81df, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:41:01.250 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:00 smithi149 podman[73680]: 2024-04-23 03:41:00.957589216 +0000 UTC m=+0.374014889 container start 79ec9ad95f45f6c5692b956af68f4c6e9e4a60b5705279577ba13adc22ce6202 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240416, RELEASE=main-56e81df, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:41:01.250 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:01 smithi149 podman[73680]: 2024-04-23 03:41:01.029317718 +0000 UTC m=+0.445743402 container attach 79ec9ad95f45f6c5692b956af68f4c6e9e4a60b5705279577ba13adc22ce6202 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, RELEASE=main-56e81df, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-23T03:41:01.532 INFO:journalctl@ceph.osd.5.smithi149.stdout:Apr 23 03:41:01 smithi149 podman[73680]: 2024-04-23 03:41:01.250520073 +0000 UTC m=+0.666945751 container died 79ec9ad95f45f6c5692b956af68f4c6e9e4a60b5705279577ba13adc22ce6202 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, RELEASE=main-56e81df) 2024-04-23T03:41:01.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:01 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:01.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:01.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:01 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:01.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:01.883 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:01 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:01.652+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:01.907 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.5.service' 2024-04-23T03:41:01.950 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:41:01.950 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-23T03:41:01.950 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-23T03:41:01.951 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.6 2024-04-23T03:41:02.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:01 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:01.910+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:02.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:01 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:01.910+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:02.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:01 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:01.910+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:02.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 systemd[1]: Stopping Ceph osd.6 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:02.660 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:02.276+0000 7f51efc2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:41:02.660 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:02.276+0000 7f51efc2d700 -1 osd.6 152 *** Got signal Terminated *** 2024-04-23T03:41:02.660 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:02.276+0000 7f51efc2d700 -1 osd.6 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:41:02.921 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:02.896+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:02.921 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:02.896+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:02.921 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:02.896+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:02.921 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:02.659+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:02.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:02.659+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:02.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:02 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:02.659+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:03.887 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:03.867+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:03.887 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:03.867+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:03.887 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:03.867+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:03.888 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:03.626+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:03.888 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:03.626+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:03.888 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:03 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:03.626+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:04.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:04 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=error ts=2024-04-23T03:41:04.242Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:41:04.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:04 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:41:04.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:41:04.582 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:04 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:41:04.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:41:04.834 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:04.614+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:04.834 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:04.614+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:04.834 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:04.614+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:04.834 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:04.614+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:05.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:04.833+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:05.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:04.833+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:05.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:04.833+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:05.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:04 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:04.833+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:38.879529+0000 front 2024-04-23T03:40:38.879557+0000 (oldest deadline 2024-04-23T03:41:04.179462+0000) 2024-04-23T03:41:05.855 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:05.593+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:05.856 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:05.593+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:05.856 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:05.593+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:05.856 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:05.593+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:06.171 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:05.854+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:06.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:05.854+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:06.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:05.854+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:06.172 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:05 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:05.854+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:38.879529+0000 front 2024-04-23T03:40:38.879557+0000 (oldest deadline 2024-04-23T03:41:04.179462+0000) 2024-04-23T03:41:06.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:06.861+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.777271+0000 front 2024-04-23T03:40:17.777349+0000 (oldest deadline 2024-04-23T03:40:43.677130+0000) 2024-04-23T03:41:06.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:06.861+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:23.677605+0000 front 2024-04-23T03:40:23.677598+0000 (oldest deadline 2024-04-23T03:40:49.577561+0000) 2024-04-23T03:41:06.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:06.861+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:32.979312+0000 front 2024-04-23T03:40:32.979267+0000 (oldest deadline 2024-04-23T03:40:58.879047+0000) 2024-04-23T03:41:06.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6[62704]: 2024-04-23T03:41:06.861+0000 7f51e8219700 -1 osd.6 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:38.879529+0000 front 2024-04-23T03:40:38.879557+0000 (oldest deadline 2024-04-23T03:41:04.179462+0000) 2024-04-23T03:41:06.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:06.622+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:06.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:06.622+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:06.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:06.622+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:06.923 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:06 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:06.622+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:07.922 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:07 smithi149 podman[73783]: 2024-04-23 03:41:07.797909118 +0000 UTC m=+5.627755683 container died ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240416, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-56e81df) 2024-04-23T03:41:07.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:07.623+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:07.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:07.623+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:07.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:07.623+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:07.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:07 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:07.623+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:08.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:08 smithi149 podman[73783]: 2024-04-23 03:41:08.251840208 +0000 UTC m=+6.081686802 container cleanup ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.vendor=CentOS, ceph=True) 2024-04-23T03:41:08.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:08 smithi149 bash[73783]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6 2024-04-23T03:41:08.853 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:08 smithi149 podman[73800]: 2024-04-23 03:41:08.519636198 +0000 UTC m=+0.717185352 container remove ec30c8d5356bad32d58728cffa8004856bb08379070e1a15937841250066936e (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, ceph=True, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-23T03:41:08.854 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:08.646+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:08.854 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:08.646+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:08.854 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:08.646+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:08.854 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:08 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:08.646+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:09.147 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:08 smithi149 podman[73889]: 2024-04-23 03:41:08.853514759 +0000 UTC m=+0.104848808 container create 7f3e894aec5565ee508ecfe6bea29c73e929d97fca215fc2ac6f17c7969bda7d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, io.buildah.version=1.33.5) 2024-04-23T03:41:09.147 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:08 smithi149 podman[73889]: 2024-04-23 03:41:08.766608336 +0000 UTC m=+0.017942409 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:41:09.147 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:09 smithi149 podman[73889]: 2024-04-23 03:41:09.072605926 +0000 UTC m=+0.323939976 container init 7f3e894aec5565ee508ecfe6bea29c73e929d97fca215fc2ac6f17c7969bda7d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-56e81df, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-23T03:41:09.148 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:09 smithi149 podman[73889]: 2024-04-23 03:41:09.075685639 +0000 UTC m=+0.327019694 container start 7f3e894aec5565ee508ecfe6bea29c73e929d97fca215fc2ac6f17c7969bda7d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240416, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-23T03:41:09.148 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:09 smithi149 podman[73889]: 2024-04-23 03:41:09.147494472 +0000 UTC m=+0.398828522 container attach 7f3e894aec5565ee508ecfe6bea29c73e929d97fca215fc2ac6f17c7969bda7d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240416, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, io.buildah.version=1.33.5) 2024-04-23T03:41:09.422 INFO:journalctl@ceph.osd.6.smithi149.stdout:Apr 23 03:41:09 smithi149 podman[73889]: 2024-04-23 03:41:09.373490205 +0000 UTC m=+0.624824265 container died 7f3e894aec5565ee508ecfe6bea29c73e929d97fca215fc2ac6f17c7969bda7d (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-56e81df, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:41:09.875 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:09.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:09.875 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:09.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:09.876 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:09.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:09.876 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:09 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:09.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:09.902 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.6.service' 2024-04-23T03:41:09.940 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:41:09.940 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-23T03:41:09.940 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-23T03:41:09.941 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.7 2024-04-23T03:41:10.171 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 systemd[1]: Stopping Ceph osd.7 for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:10.671 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.269+0000 7f3e3604b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-23T03:41:10.672 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.269+0000 7f3e3604b700 -1 osd.7 152 *** Got signal Terminated *** 2024-04-23T03:41:10.672 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.269+0000 7f3e3604b700 -1 osd.7 152 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-23T03:41:11.173 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.694+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:11.173 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.694+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:11.173 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.694+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:11.174 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:10 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:10.694+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:12.073 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:11.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:12.073 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:11.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:12.074 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:11.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:12.074 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:11 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:11.696+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:12.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:12.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:12.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:12.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:12.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:12.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:12.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:12 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:12.670+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:13.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:13.624+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:13.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:13.624+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:13.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:13.624+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:13.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:13 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:13.624+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:14.506 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=error ts=2024-04-23T03:41:14.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:41:14.507 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:41:14.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi007.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.7:8443: connect: connection refused" 2024-04-23T03:41:14.507 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:14 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=warn ts=2024-04-23T03:41:14.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi149:8443/api/prometheus_receiver\": dial tcp 172.21.15.149:8443: connect: connection refused" 2024-04-23T03:41:14.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:14.650+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-23T03:40:17.215038+0000 front 2024-04-23T03:40:17.215118+0000 (oldest deadline 2024-04-23T03:40:43.115094+0000) 2024-04-23T03:41:14.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:14.650+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-23T03:40:24.216148+0000 front 2024-04-23T03:40:24.216259+0000 (oldest deadline 2024-04-23T03:40:47.716016+0000) 2024-04-23T03:41:14.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:14.650+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-23T03:40:31.217057+0000 front 2024-04-23T03:40:31.217098+0000 (oldest deadline 2024-04-23T03:40:55.916931+0000) 2024-04-23T03:41:14.922 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:14 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7[65064]: 2024-04-23T03:41:14.650+0000 7f3e2e637700 -1 osd.7 152 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-23T03:40:41.817911+0000 front 2024-04-23T03:40:41.817934+0000 (oldest deadline 2024-04-23T03:41:04.117842+0000) 2024-04-23T03:41:15.863 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:15 smithi149 podman[73995]: 2024-04-23 03:41:15.600418729 +0000 UTC m=+5.443510470 container died 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240416, RELEASE=main-56e81df, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5) 2024-04-23T03:41:16.171 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:15 smithi149 podman[73995]: 2024-04-23 03:41:15.928140031 +0000 UTC m=+5.771231768 container cleanup 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240416, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-23T03:41:16.172 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:15 smithi149 bash[73995]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7 2024-04-23T03:41:16.172 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:16 smithi149 podman[74024]: 2024-04-23 03:41:16.095812553 +0000 UTC m=+0.493853529 container remove 7a072e4b062d718e1faaa3cb7ebadf809ef0792b8454c5720d7480f7ecb8c965 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7, RELEASE=main-56e81df, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-23T03:41:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:16 smithi149 podman[74116]: 2024-04-23 03:41:16.538136011 +0000 UTC m=+0.110268802 container create 531e3c7bb9031694d7c86e09f1141b7bf82a20358831f34086a0bbeb21a9a7d8 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-56e81df, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-23T03:41:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:16 smithi149 podman[74116]: 2024-04-23 03:41:16.445853229 +0000 UTC m=+0.017986035 image pull 82086b19bb070a73a22a9572b4d46af73014eba1e8e605aa289115cdc1fd1de4 quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7 2024-04-23T03:41:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:16 smithi149 podman[74116]: 2024-04-23 03:41:16.749114629 +0000 UTC m=+0.321247433 container init 531e3c7bb9031694d7c86e09f1141b7bf82a20358831f34086a0bbeb21a9a7d8 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240416, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-56e81df, ceph=True) 2024-04-23T03:41:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:16 smithi149 podman[74116]: 2024-04-23 03:41:16.752280073 +0000 UTC m=+0.324412865 container start 531e3c7bb9031694d7c86e09f1141b7bf82a20358831f34086a0bbeb21a9a7d8 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240416, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=main-56e81df, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True) 2024-04-23T03:41:16.849 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:16 smithi149 podman[74116]: 2024-04-23 03:41:16.848839319 +0000 UTC m=+0.420972111 container attach 531e3c7bb9031694d7c86e09f1141b7bf82a20358831f34086a0bbeb21a9a7d8 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240416, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=main-56e81df) 2024-04-23T03:41:17.171 INFO:journalctl@ceph.osd.7.smithi149.stdout:Apr 23 03:41:17 smithi149 podman[74116]: 2024-04-23 03:41:17.043724467 +0000 UTC m=+0.615857277 container died 531e3c7bb9031694d7c86e09f1141b7bf82a20358831f34086a0bbeb21a9a7d8 (image=quay.ceph.io/ceph-ci/ceph:56e81df3aeb98b717efe2ab0537a8c60249f95f7, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=main-56e81df, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240416, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True) 2024-04-23T03:41:17.649 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@osd.7.service' 2024-04-23T03:41:17.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:41:17.688 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-23T03:41:17.688 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-23T03:41:17.688 DEBUG:teuthology.orchestra.run.smithi149:> sudo systemctl stop ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a 2024-04-23T03:41:17.882 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 systemd[1]: Stopping Ceph prometheus.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:17.882 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:41:17.882 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.871Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.872Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-23T03:41:17.883 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a[50744]: ts=2024-04-23T03:41:17.872Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-23T03:41:17.884 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:17 smithi149 podman[74196]: 2024-04-23 03:41:17.881651249 +0000 UTC m=+0.106376155 container died 86ffd466d290094f1166865f0a5315bd3ac67ab1fe4bca00d5d73dccd43e45fc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:18.421 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:18 smithi149 podman[74196]: 2024-04-23 03:41:18.220830614 +0000 UTC m=+0.445555535 container cleanup 86ffd466d290094f1166865f0a5315bd3ac67ab1fe4bca00d5d73dccd43e45fc (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:18.421 INFO:journalctl@ceph.prometheus.a.smithi149.stdout:Apr 23 03:41:18 smithi149 bash[74196]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-prometheus-a 2024-04-23T03:41:18.499 DEBUG:teuthology.orchestra.run.smithi149:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3d05926e-0120-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-23T03:41:18.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-23T03:41:18.532 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-23T03:41:18.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d05926e-0120-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T03:41:20.332 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:20 smithi007 systemd[1]: Stopping Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:20.600 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:20 smithi007 podman[89619]: 2024-04-23 03:41:20.334690522 +0000 UTC m=+0.125154943 container died 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:21.076 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:20 smithi007 podman[89619]: 2024-04-23 03:41:20.699024954 +0000 UTC m=+0.489489368 container cleanup 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:21.076 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:20 smithi007 bash[89619]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a 2024-04-23T03:41:21.076 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:20 smithi007 bash[89645]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-23T03:41:21.332 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:21 smithi007 podman[89633]: 2024-04-23 03:41:21.076193155 +0000 UTC m=+0.745371164 container remove 45fdbfb255451417049c56a3fb7a190ce5f5c7e6d0cca66d0033dc569005095f (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:21.332 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:21 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:41:21.332 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:21 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-23T03:41:21.332 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:21 smithi007 systemd[1]: Stopped Ceph node-exporter.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:41:21.333 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 23 03:41:21 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 3.729s CPU time. 2024-04-23T03:41:21.333 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:21.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a[56292]: level=info ts=2024-04-23T03:41:21.528Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-23T03:41:21.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 podman[89699]: 2024-04-23 03:41:21.569397904 +0000 UTC m=+0.283266365 container stop dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:21.832 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 podman[89699]: 2024-04-23 03:41:21.594913282 +0000 UTC m=+0.308781732 container died dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:22.204 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 podman[89699]: 2024-04-23 03:41:21.936503744 +0000 UTC m=+0.650372195 container cleanup dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:22.204 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 bash[89699]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a 2024-04-23T03:41:22.204 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:21 smithi007 bash[89725]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-23T03:41:22.478 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:22 smithi007 podman[89713]: 2024-04-23 03:41:22.204230414 +0000 UTC m=+0.625009146 container remove dd67074f48e470fb4899aa24c22441bdfefd257ee9953617b8c6cd788c15160a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-23T03:41:22.478 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:22 smithi007 podman[89713]: 2024-04-23 03:41:22.279857537 +0000 UTC m=+0.700636243 volume remove c8135ff41fd7feacfb1aa6b6a909e1688a750075c7f881d27c26cf2dc37848ae 2024-04-23T03:41:22.478 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:22 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-23T03:41:22.478 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:22 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:41:22.478 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 23 03:41:22 smithi007 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@alertmanager.a.service: Consumed 1.010s CPU time. 2024-04-23T03:41:56.470 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d05926e-0120-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-23T03:41:58.199 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:57 smithi149 systemd[1]: Stopping Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:58.199 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 podman[74554]: 2024-04-23 03:41:58.048200537 +0000 UTC m=+0.112744337 container died 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:41:58.456 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 podman[74554]: 2024-04-23 03:41:58.420738779 +0000 UTC m=+0.485282587 container cleanup 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:41:58.456 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 bash[74554]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b 2024-04-23T03:41:58.456 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 bash[74582]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-23T03:41:58.922 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:58 smithi149 systemd[1]: Stopping Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968... 2024-04-23T03:41:58.922 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 podman[74569]: 2024-04-23 03:41:58.638625388 +0000 UTC m=+0.594233066 container remove 2d32c60dbd33c5703ab6259cc04743e4034f7de42d035cb121edee84f73dd6b2 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-23T03:41:58.922 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-23T03:41:58.922 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-23T03:41:58.923 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 systemd[1]: Stopped Ceph node-exporter.b for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:41:58.923 INFO:journalctl@ceph.node-exporter.b.smithi149.stdout:Apr 23 03:41:58 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 3.649s CPU time. 2024-04-23T03:41:59.241 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:58 smithi149 ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a[47959]: t=2024-04-23T03:41:58+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-23T03:41:59.241 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:58 smithi149 podman[74646]: 2024-04-23 03:41:58.934878099 +0000 UTC m=+0.096718162 container died 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , build-date=2022-03-28T10:36:18.413762, com.redhat.component=ubi8-container, version=8.5, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., architecture=x86_64, release=236.1648460182, io.buildah.version=1.24.2, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git) 2024-04-23T03:41:59.241 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 podman[74646]: 2024-04-23 03:41:59.241312084 +0000 UTC m=+0.403152151 container cleanup 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, architecture=x86_64, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.tags=base rhel8, release=236.1648460182, distribution-scope=public, io.buildah.version=1.24.2, maintainer=Paul Cuzner , vendor=Red Hat, Inc., build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git) 2024-04-23T03:41:59.504 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 bash[74646]: ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a 2024-04-23T03:41:59.504 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 bash[74672]: Error: no container with name or ID "ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-23T03:41:59.504 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 podman[74660]: 2024-04-23 03:41:59.484344323 +0000 UTC m=+0.550582906 container remove 4865bb18d5e14c877745114466e1d2b4ea4b98a16f1a6f9a4d5776d793c39978 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-3d05926e-0120-11ef-bc93-c7b262605968-grafana-a, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, name=ubi8, build-date=2022-03-28T10:36:18.413762, io.openshift.tags=base rhel8, architecture=x86_64, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, release=236.1648460182, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , vendor=Red Hat, Inc.) 2024-04-23T03:41:59.504 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-23T03:41:59.504 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 systemd[1]: Stopped Ceph grafana.a for 3d05926e-0120-11ef-bc93-c7b262605968. 2024-04-23T03:41:59.505 INFO:journalctl@ceph.grafana.a.smithi149.stdout:Apr 23 03:41:59 smithi149 systemd[1]: ceph-3d05926e-0120-11ef-bc93-c7b262605968@grafana.a.service: Consumed 4.034s CPU time. 2024-04-23T03:42:22.446 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:42:22.482 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-23T03:42:22.508 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-23T03:42:22.510 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/remote/smithi007/crash 2024-04-23T03:42:22.511 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/crash -- . 2024-04-23T03:42:22.560 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T03:42:22.560 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-04-23T03:42:22.563 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/remote/smithi149/crash 2024-04-23T03:42:22.564 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/crash -- . 2024-04-23T03:42:22.600 INFO:teuthology.orchestra.run.smithi149.stderr:tar: /var/lib/ceph/3d05926e-0120-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-23T03:42:22.600 INFO:teuthology.orchestra.run.smithi149.stderr:tar: Error is not recoverable: exiting now 2024-04-23T03:42:22.601 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-23T03:42:22.601 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-23T03:42:22.633 INFO:teuthology.orchestra.run.smithi007.stdout:2024-04-23T03:31:41.459718+0000 mon.a (mon.0) 1038 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:42:22.633 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-23T03:42:22.634 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[SEC\]' /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-23T03:42:22.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]' /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-23T03:42:22.773 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[WRN\]' /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-23T03:42:22.842 INFO:teuthology.orchestra.run.smithi007.stdout:2024-04-23T03:31:41.459718+0000 mon.a (mon.0) 1038 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-23T03:42:22.842 INFO:tasks.cephadm:Compressing logs... 2024-04-23T03:42:22.843 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T03:42:22.885 DEBUG:teuthology.orchestra.run.smithi149:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-23T03:42:22.910 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T03:42:22.910 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-23T03:42:22.911 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-23T03:42:22.913 INFO:teuthology.orchestra.run.smithi149.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-23T03:42:22.913 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-23T03:42:22.913 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log 2024-04-23T03:42:22.913 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-23T03:42:22.913 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.7.log/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log: 2024-04-23T03:42:22.913 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-23T03:42:22.914 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-23T03:42:22.914 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-23T03:42:22.915 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mgr.y.log: 92.9% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T03:42:22.915 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-23T03:42:22.916 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.7.log: /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.4.log: /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.6.log: 87.2% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi149.axrzpl.log 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.cephadm.log: 85.8% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi149.axrzpl.log: 90.5% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr: 76.8% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi149.axrzpl.log.gzgzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-04-23T03:42:22.917 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi149.rtdeuy.log 2024-04-23T03:42:22.918 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.0.log: 94.2% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-23T03:42:22.919 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi007.mxntqi.log 2024-04-23T03:42:22.921 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.c.log: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T03:42:22.928 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T03:42:22.931 INFO:teuthology.orchestra.run.smithi149.stderr: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-23T03:42:22.931 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-23T03:42:22.932 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi007.mxntqi.log: 76.3% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi007.mxntqi.log.gz 2024-04-23T03:42:22.940 INFO:teuthology.orchestra.run.smithi149.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi149.rtdeuy.log: 76.8% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi149.rtdeuy.log.gz 2024-04-23T03:42:22.944 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi007.bipufo.log 2024-04-23T03:42:22.956 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-23T03:42:22.957 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi007.bipufo.log: 76.0% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi007.bipufo.log.gz 2024-04-23T03:42:22.964 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-23T03:42:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-23T03:42:22.976 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/tcmu-runner.log: 84.4% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-23T03:42:22.978 INFO:teuthology.orchestra.run.smithi007.stderr: 91.8% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-23T03:42:22.978 INFO:teuthology.orchestra.run.smithi149.stderr: 89.6% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-23T03:42:23.014 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-23T03:42:23.275 INFO:teuthology.orchestra.run.smithi007.stderr: 88.5% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-23T03:42:23.357 INFO:teuthology.orchestra.run.smithi149.stderr: 92.4% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-23T03:42:23.468 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-23T03:42:23.929 INFO:teuthology.orchestra.run.smithi149.stderr: 93.6% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-23T03:42:24.093 INFO:teuthology.orchestra.run.smithi007.stderr: 93.7% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-23T03:42:24.118 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-23T03:42:24.298 INFO:teuthology.orchestra.run.smithi149.stderr: 93.8% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-23T03:42:24.398 INFO:teuthology.orchestra.run.smithi149.stderr: 93.6% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-23T03:42:24.398 INFO:teuthology.orchestra.run.smithi007.stderr: 93.7% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-23T03:42:24.503 INFO:teuthology.orchestra.run.smithi149.stderr: 93.8% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-23T03:42:24.505 INFO:teuthology.orchestra.run.smithi149.stderr: 2024-04-23T03:42:24.505 INFO:teuthology.orchestra.run.smithi149.stderr:real 0m1.605s 2024-04-23T03:42:24.505 INFO:teuthology.orchestra.run.smithi149.stderr:user 0m5.819s 2024-04-23T03:42:24.505 INFO:teuthology.orchestra.run.smithi149.stderr:sys 0m0.228s 2024-04-23T03:42:24.725 INFO:teuthology.orchestra.run.smithi007.stderr: 93.7% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-23T03:42:24.758 INFO:teuthology.orchestra.run.smithi007.stderr: 93.7% -- replaced with /var/log/ceph/3d05926e-0120-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-23T03:42:24.760 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-23T03:42:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m1.862s 2024-04-23T03:42:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m8.175s 2024-04-23T03:42:24.760 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.270s 2024-04-23T03:42:24.761 INFO:tasks.cephadm:Archiving logs... 2024-04-23T03:42:24.762 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/remote/smithi007/log 2024-04-23T03:42:24.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T03:42:25.696 DEBUG:teuthology.misc:Transferring archived files from smithi149:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/remote/smithi149/log 2024-04-23T03:42:25.697 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-23T03:42:26.458 INFO:tasks.cephadm:Removing cluster... 2024-04-23T03:42:26.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d05926e-0120-11ef-bc93-c7b262605968 --force 2024-04-23T03:42:26.757 DEBUG:teuthology.orchestra.run.smithi149:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 3d05926e-0120-11ef-bc93-c7b262605968 --force 2024-04-23T03:42:27.072 INFO:tasks.cephadm:Removing cephadm ... 2024-04-23T03:42:27.072 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T03:42:27.088 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-23T03:42:27.104 INFO:tasks.cephadm:Teardown complete 2024-04-23T03:42:27.105 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-23T03:42:27.116 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-23T03:42:27.116 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T03:42:27.131 DEBUG:teuthology.orchestra.run.smithi149:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-23T03:42:27.143 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-04-23T03:42:27.149 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T03:42:27.149 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-04-23T03:42:27.149 INFO:teuthology.orchestra.run.smithi007.stdout:^- hv01.front.sepia.ceph.com 2 6 377 40 +717us[ +729us] +/- 610ms 2024-04-23T03:42:27.149 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 7 377 37 -1079us[-1067us] +/- 74ms 2024-04-23T03:42:27.149 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1276us[+1289us] +/- 70ms 2024-04-23T03:42:27.149 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:42:27.158 INFO:teuthology.orchestra.run.smithi149.stderr:bash: line 1: ntpq: command not found 2024-04-23T03:42:27.163 INFO:teuthology.orchestra.run.smithi149.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-23T03:42:27.164 INFO:teuthology.orchestra.run.smithi149.stdout:=============================================================================== 2024-04-23T03:42:27.164 INFO:teuthology.orchestra.run.smithi149.stdout:^- hv01.front.sepia.ceph.com 2 6 377 39 +590us[ +590us] +/- 610ms 2024-04-23T03:42:27.164 INFO:teuthology.orchestra.run.smithi149.stdout:^* hv02.front.sepia.ceph.com 3 7 377 103 -1214us[-1107us] +/- 73ms 2024-04-23T03:42:27.164 INFO:teuthology.orchestra.run.smithi149.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +1165us[+1165us] +/- 70ms 2024-04-23T03:42:27.164 INFO:teuthology.orchestra.run.smithi149.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-23T03:42:27.164 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-23T03:42:27.174 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-23T03:42:27.175 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-23T03:42:27.184 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T03:42:27.268 DEBUG:teuthology.orchestra.run.smithi149:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-23T03:42:27.339 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-23T03:42:27.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:42:27.368 DEBUG:teuthology.orchestra.run.smithi149:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-23T03:42:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:42:27.396 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-23T03:42:27.406 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-23T03:42:27.415 INFO:teuthology.task.internal:Duration was 1613.783653 seconds 2024-04-23T03:42:27.416 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-23T03:42:27.425 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-23T03:42:27.425 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T03:42:27.427 DEBUG:teuthology.orchestra.run.smithi149:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-23T03:42:27.463 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T03:42:27.474 INFO:teuthology.orchestra.run.smithi149.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-23T03:42:27.708 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-23T03:42:27.708 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-04-23T03:42:27.708 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T03:42:27.729 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi149.front.sepia.ceph.com 2024-04-23T03:42:27.730 DEBUG:teuthology.orchestra.run.smithi149:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-23T03:42:27.774 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-23T03:42:27.774 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T03:42:27.777 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-23T03:42:27.839 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-23T03:42:27.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T03:42:27.843 DEBUG:teuthology.orchestra.run.smithi149:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-23T03:42:28.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-23T03:42:28.313 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-23T03:42:28.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T03:42:28.338 DEBUG:teuthology.orchestra.run.smithi149:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-23T03:42:28.363 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-23T03:42:28.373 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T03:42:28.382 DEBUG:teuthology.orchestra.run.smithi149:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-23T03:42:28.402 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-04-23T03:42:28.425 INFO:teuthology.orchestra.run.smithi149.stdout:kernel.core_pattern = core 2024-04-23T03:42:28.476 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T03:42:28.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:42:28.491 DEBUG:teuthology.orchestra.run.smithi149:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-23T03:42:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-23T03:42:28.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-23T03:42:28.540 INFO:teuthology.task.internal:Transferring archived files... 2024-04-23T03:42:28.541 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/remote/smithi007 2024-04-23T03:42:28.542 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T03:42:28.588 DEBUG:teuthology.misc:Transferring archived files from smithi149:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734/remote/smithi149 2024-04-23T03:42:28.589 DEBUG:teuthology.orchestra.run.smithi149:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-23T03:42:28.628 INFO:teuthology.task.internal:Removing archive directory... 2024-04-23T03:42:28.628 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T03:42:28.630 DEBUG:teuthology.orchestra.run.smithi149:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-23T03:42:28.674 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-23T03:42:28.685 INFO:teuthology.task.internal:Not uploading archives. 2024-04-23T03:42:28.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-23T03:42:28.695 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-23T03:42:28.695 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T03:42:28.698 DEBUG:teuthology.orchestra.run.smithi149:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-23T03:42:28.710 INFO:teuthology.orchestra.run.smithi007.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 03:42 /home/ubuntu/cephtest 2024-04-23T03:42:28.728 INFO:teuthology.orchestra.run.smithi149.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 23 03:42 /home/ubuntu/cephtest 2024-04-23T03:42:28.729 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-23T03:42:28.739 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-23T03:42:28.760 INFO:teuthology.nuke:Checking targets against current locks 2024-04-23T03:42:28.789 DEBUG:teuthology.nuke:shortname: smithi007 2024-04-23T03:42:28.789 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T03:42:28.813 DEBUG:teuthology.nuke:shortname: smithi149 2024-04-23T03:42:28.813 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-23T03:42:28.817 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 03:09:34.868345', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEjLpAwTZqGo+f69gG/J23ZYHahwEZJxuD/yaed9FhB297Avs9YMRhGQ+kf1nRdmGz4PSGxcunfzdv2PPBgcrb8='} 2024-04-23T03:42:28.835 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi149.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_20:08:13-orch-main-distro-default-smithi/7668734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-23 03:09:34.870034', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCE/dledikhDHGIDwcvDkB4g9yGeG5dsOjTgrNpdLsrc4ksWDm22WN6lerGXkKkYEvqMPVA8XtBiuN6292PY8PA='} 2024-04-23T03:42:28.866 INFO:teuthology.orchestra.console.smithi007:Power off 2024-04-23T03:42:28.866 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T03:42:28.895 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-04-23T03:42:28.895 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T03:42:28.909 INFO:teuthology.orchestra.console.smithi149:Power off 2024-04-23T03:42:28.909 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-23T03:42:28.918 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-04-23T03:42:28.930 DEBUG:teuthology.orchestra.console.smithi149:power off output: Chassis Power Control: Down/Off 2024-04-23T03:42:28.930 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T03:42:28.953 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-04-23T03:42:32.920 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T03:42:32.954 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T03:42:33.030 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-04-23T03:42:33.064 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is on 2024-04-23T03:42:37.034 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T03:42:37.065 DEBUG:teuthology.orchestra.console.smithi149:pexpect command: ipmitool -H smithi149.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-23T03:42:37.143 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-04-23T03:42:37.172 DEBUG:teuthology.orchestra.console.smithi149:check power output: Chassis Power is off 2024-04-23T03:42:37.244 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-04-23T03:42:37.273 INFO:teuthology.orchestra.console.smithi149:Power off completed 2024-04-23T03:42:37.376 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 1613.7836525440216 failure_reason: '"2024-04-23T03:31:41.459718+0000 mon.a (mon.0) 1038 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-04-23T03:42:37.377 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-23T03:42:37.486 INFO:teuthology.run:FAIL