2024-06-01T02:23:09.710 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-01T02:23:09.710 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} --archive /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417 --owner scheduled_teuthology@teuthology --name teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/orig.config.yaml 2024-06-01T02:23:09.742 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-01T02:23:09.814 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7736417' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba workunit: branch: squid sha1: 6dca6333905c2b38b02396dfd1a76a97d29c2ee7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 3011 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 21/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_6dca6333905c2b38b02396dfd1a76a97d29c2ee7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 6dca6333905c2b38b02396dfd1a76a97d29c2ee7 targets: smithi062.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL92fpOBYQyh4w7S8dkY6dmOzW1xiz+JliqDuhiFETe+JteVhUqxZyHlj+th/RUKwTIN7IHKipxMQdv9lhYz6V0= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdODRJODLi2ut9IwL62R+0keYuOtddVRQozxGzoRvldv+WrNwgRfXmenU7U/rkgbDU/rccfOs05iaAfHNJbi2Y= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-05-31_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-01T02:23:09.814 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_6dca6333905c2b38b02396dfd1a76a97d29c2ee7/qa; will attempt to use it 2024-06-01T02:23:09.815 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_6dca6333905c2b38b02396dfd1a76a97d29c2ee7/qa/tasks 2024-06-01T02:23:09.815 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-01T02:23:09.816 INFO:teuthology.task.internal:Checking packages... 2024-06-01T02:23:09.838 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-01T02:23:09.838 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-01T02:23:09.838 INFO:teuthology.packaging:ref: None 2024-06-01T02:23:09.838 INFO:teuthology.packaging:tag: None 2024-06-01T02:23:09.838 INFO:teuthology.packaging:branch: squid 2024-06-01T02:23:09.838 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:23:09.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-01T02:23:10.056 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2946.g3d89accf 2024-06-01T02:23:10.057 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-01T02:23:10.064 INFO:teuthology.task.internal:no buildpackages task found 2024-06-01T02:23:10.064 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-01T02:23:10.081 INFO:teuthology.task.internal:Saving configuration 2024-06-01T02:23:10.093 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-01T02:23:10.100 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-01T02:23:10.123 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi062.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-01 02:18:05.999773', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL92fpOBYQyh4w7S8dkY6dmOzW1xiz+JliqDuhiFETe+JteVhUqxZyHlj+th/RUKwTIN7IHKipxMQdv9lhYz6V0='} 2024-06-01T02:23:10.144 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-01 02:18:05.997793', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPdODRJODLi2ut9IwL62R+0keYuOtddVRQozxGzoRvldv+WrNwgRfXmenU7U/rkgbDU/rccfOs05iaAfHNJbi2Y='} 2024-06-01T02:23:10.144 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-01T02:23:10.150 INFO:teuthology.task.internal:roles: ubuntu@smithi062.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-06-01T02:23:10.150 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-06-01T02:23:10.150 INFO:teuthology.run_tasks:Running task console_log... 2024-06-01T02:23:10.208 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f63d58cbac0>, signals=[15]) 2024-06-01T02:23:10.208 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-01T02:23:10.215 INFO:teuthology.task.internal:Opening connections... 2024-06-01T02:23:10.215 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi062.front.sepia.ceph.com 2024-06-01T02:23:10.216 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:23:10.308 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-06-01T02:23:10.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:23:10.384 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-01T02:23:10.391 DEBUG:teuthology.orchestra.run.smithi062:> uname -m 2024-06-01T02:23:10.409 INFO:teuthology.orchestra.run.smithi062.stdout:x86_64 2024-06-01T02:23:10.409 DEBUG:teuthology.orchestra.run.smithi062:> cat /etc/os-release 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:NAME="CentOS Stream" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION="9" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:ID="centos" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:ID_LIKE="rhel fedora" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:VERSION_ID="9" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:PLATFORM_ID="platform:el9" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-01T02:23:10.467 INFO:teuthology.orchestra.run.smithi062.stdout:ANSI_COLOR="0;31" 2024-06-01T02:23:10.468 INFO:teuthology.orchestra.run.smithi062.stdout:LOGO="fedora-logo-icon" 2024-06-01T02:23:10.468 INFO:teuthology.orchestra.run.smithi062.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-01T02:23:10.468 INFO:teuthology.orchestra.run.smithi062.stdout:HOME_URL="https://centos.org/" 2024-06-01T02:23:10.468 INFO:teuthology.orchestra.run.smithi062.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-01T02:23:10.468 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-01T02:23:10.468 INFO:teuthology.orchestra.run.smithi062.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-01T02:23:10.468 INFO:teuthology.lock.ops:Updating smithi062.front.sepia.ceph.com on lock server 2024-06-01T02:23:10.491 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-06-01T02:23:10.509 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-06-01T02:23:10.509 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-06-01T02:23:10.566 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-01T02:23:10.567 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-06-01T02:23:10.567 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-01T02:23:10.567 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-01T02:23:10.567 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-01T02:23:10.567 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-06-01T02:23:10.591 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-01T02:23:10.600 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-01T02:23:10.605 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-01T02:23:10.605 DEBUG:teuthology.orchestra.run.smithi062:> test '!' -e /home/ubuntu/cephtest 2024-06-01T02:23:10.607 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-06-01T02:23:10.623 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-01T02:23:10.629 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-01T02:23:10.629 DEBUG:teuthology.orchestra.run.smithi062:> test -z $(ls -A /var/lib/ceph) 2024-06-01T02:23:10.665 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-06-01T02:23:10.695 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-01T02:23:10.742 INFO:teuthology.run_tasks:Running task kernel... 2024-06-01T02:23:10.757 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-01T02:23:10.757 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-01T02:23:10.757 DEBUG:teuthology.orchestra.run.smithi062:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-01T02:23:10.758 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-01T02:23:10.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:23:10.774 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-06-01T02:23:10.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:23:10.775 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-06-01T02:23:10.831 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-437.el9.x86_64 2024-06-01T02:23:10.831 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.14.0-437.el9.x86_64 2024-06-01T02:23:10.831 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum install -y kernel 2024-06-01T02:23:10.834 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-437.el9.x86_64 2024-06-01T02:23:10.834 INFO:teuthology.task.kernel:Running kernel on smithi062: 5.14.0-437.el9.x86_64 2024-06-01T02:23:10.834 DEBUG:teuthology.orchestra.run.smithi062:> sudo yum install -y kernel 2024-06-01T02:23:14.315 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-06-01T02:23:15.193 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.1 MB 00:02 2024-06-01T02:23:17.027 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-06-01T02:23:23.628 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - CRB 2.2 MB/s | 6.2 MB 00:02 2024-06-01T02:23:25.103 INFO:teuthology.orchestra.run.smithi062.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 17 kB 00:00 2024-06-01T02:23:26.490 INFO:teuthology.orchestra.run.smithi062.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-06-01T02:23:31.122 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-06-01T02:23:31.838 INFO:teuthology.orchestra.run.smithi062.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-06-01T02:23:33.911 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-01T02:23:33.911 INFO:teuthology.orchestra.run.smithi062.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-01T02:23:34.009 INFO:teuthology.orchestra.run.smithi062.stdout:Dependencies resolved. 2024-06-01T02:23:34.017 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: Package Architecture Version Repository Size 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout:Installing: 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout:Installing dependencies: 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-01T02:23:34.018 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction Summary 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout:================================================================================ 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout:Install 4 Packages 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout:Total download size: 80 M 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout:Installed size: 124 M 2024-06-01T02:23:34.019 INFO:teuthology.orchestra.run.smithi062.stdout:Downloading Packages: 2024-06-01T02:23:35.006 INFO:teuthology.orchestra.run.smithi062.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 1.0 MB/s | 893 kB 00:00 2024-06-01T02:23:35.240 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-06-01T02:23:36.865 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 17 kB 00:00 2024-06-01T02:23:38.183 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-06-01T02:23:43.433 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-06-01T02:23:43.539 INFO:teuthology.orchestra.run.smithi062.stdout:(2/4): kernel-modules-core-5.14.0-452.el9.x86_6 3.4 MB/s | 29 MB 00:08 2024-06-01T02:23:45.491 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-01T02:23:45.491 INFO:teuthology.orchestra.run.smithi146.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-01T02:23:45.581 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout: Package Architecture Version Repository Size 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-01T02:23:45.591 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:23:45.592 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-06-01T02:23:45.592 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-06-01T02:23:45.592 INFO:teuthology.orchestra.run.smithi146.stdout:Install 4 Packages 2024-06-01T02:23:45.592 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:23:45.592 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 80 M 2024-06-01T02:23:45.592 INFO:teuthology.orchestra.run.smithi146.stdout:Installed size: 124 M 2024-06-01T02:23:45.593 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-06-01T02:23:45.853 INFO:teuthology.orchestra.run.smithi146.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 4.4 MB/s | 893 kB 00:00 2024-06-01T02:23:46.520 INFO:teuthology.orchestra.run.smithi146.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-06-01T02:23:46.954 INFO:teuthology.orchestra.run.smithi146.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-06-01T02:23:47.295 INFO:teuthology.orchestra.run.smithi146.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 20 MB/s | 29 MB 00:01 2024-06-01T02:23:47.296 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-06-01T02:23:47.296 INFO:teuthology.orchestra.run.smithi146.stdout:Total 47 MB/s | 80 MB 00:01 2024-06-01T02:23:47.872 INFO:teuthology.orchestra.run.smithi062.stdout:(3/4): kernel-core-5.14.0-452.el9.x86_64.rpm 1.2 MB/s | 16 MB 00:13 2024-06-01T02:23:47.995 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-06-01T02:23:48.344 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-06-01T02:23:48.344 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-06-01T02:23:49.302 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-06-01T02:23:49.302 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-06-01T02:23:51.740 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-06-01T02:23:52.467 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-01T02:23:52.554 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T02:23:53.972 INFO:teuthology.orchestra.run.smithi062.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 1.7 MB/s | 35 MB 00:19 2024-06-01T02:23:53.972 INFO:teuthology.orchestra.run.smithi062.stdout:-------------------------------------------------------------------------------- 2024-06-01T02:23:53.972 INFO:teuthology.orchestra.run.smithi062.stdout:Total 4.0 MB/s | 80 MB 00:19 2024-06-01T02:23:54.699 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction check 2024-06-01T02:23:55.102 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction check succeeded. 2024-06-01T02:23:55.102 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction test 2024-06-01T02:23:55.193 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T02:23:55.339 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T02:23:56.067 INFO:teuthology.orchestra.run.smithi062.stdout:Transaction test succeeded. 2024-06-01T02:23:56.067 INFO:teuthology.orchestra.run.smithi062.stdout:Running transaction 2024-06-01T02:23:58.669 INFO:teuthology.orchestra.run.smithi062.stdout: Preparing : 1/1 2024-06-01T02:23:59.341 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-01T02:23:59.429 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T02:24:00.114 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T02:24:00.385 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:02.225 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T02:24:02.455 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T02:24:05.172 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:07.548 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T02:24:07.610 INFO:teuthology.orchestra.run.smithi062.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:12.865 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:34.764 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:34.803 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:35.178 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:35.178 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-01T02:24:35.178 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T02:24:35.178 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T02:24:35.959 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:24:35.960 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-06-01T02:24:36.300 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T02:24:36.379 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T02:24:36.379 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T02:24:36.379 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-01T02:24:36.379 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-01T02:24:36.379 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-01T02:24:36.380 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-01T02:24:36.380 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-06-01T02:24:36.380 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T02:24:36.431 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T02:24:36.432 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T02:24:36.432 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-06-01T02:24:36.463 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-01T02:24:36.464 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-01T02:24:36.465 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-06-01T02:24:36.466 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:24:36.466 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-06-01T02:24:36.467 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-01T02:24:36.467 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 5.14.0-452.el9.x86_64 2024-06-01T02:24:36.467 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-01T02:24:37.064 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-06-01T02:24:38.434 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-01T02:24:38.463 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-06-01T02:24:38.466 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-06-01T02:24:38.481 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.ehyVCybwPE 2024-06-01T02:24:38.482 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ehyVCybwPE 2024-06-01T02:24:38.548 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.ehyVCybwPE 2024-06-01T02:24:38.683 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.ehyVCybwPE is 6KB 2024-06-01T02:24:38.732 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.ehyVCybwPE 2024-06-01T02:24:38.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-06-01T02:24:38.815 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-01T02:24:38.815 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-01T02:24:38.815 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-01T02:24:38.816 INFO:teuthology.orchestra.run.smithi146.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-01T02:24:38.817 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-01T02:24:38.961 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-06-01T02:24:43.660 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:43.701 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:44.055 INFO:teuthology.orchestra.run.smithi062.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:44.055 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-01T02:24:44.055 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-01T02:24:44.055 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout:Installed: 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:24:44.804 INFO:teuthology.orchestra.run.smithi062.stdout:Complete! 2024-06-01T02:24:45.159 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T02:24:45.258 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T02:24:45.258 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T02:24:45.259 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-01T02:24:45.259 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-01T02:24:45.259 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-01T02:24:45.259 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-01T02:24:45.259 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi062.front.sepia.ceph.com, path=None, version=distro) 2024-06-01T02:24:45.259 DEBUG:teuthology.orchestra.run.smithi062:> rpm -q kernel | sort -rV | head -n 1 2024-06-01T02:24:45.312 INFO:teuthology.orchestra.run.smithi062.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-01T02:24:45.313 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-01T02:24:45.313 DEBUG:teuthology.orchestra.run.smithi062:> sudo rpm -qi grub2-tools 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Name : grub2-tools 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Epoch : 1 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Version : 2.06 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Release : 68.el9 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Architecture: x86_64 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Group : Unspecified 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Size : 8264254 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:License : GPLv3+ 2024-06-01T02:24:45.355 INFO:teuthology.orchestra.run.smithi062.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Packager : builder@centos.org 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Vendor : CentOS 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Summary : Support tools for GRUB. 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:Description : 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-01T02:24:45.356 INFO:teuthology.orchestra.run.smithi062.stdout:hardware devices. 2024-06-01T02:24:45.357 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:24:45.357 INFO:teuthology.orchestra.run.smithi062.stdout:This subpackage provides tools for support of all platforms. 2024-06-01T02:24:45.357 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-01T02:24:45.358 INFO:teuthology.task.kernel:Updating grub on smithi062 to boot 5.14.0-452.el9.x86_64 2024-06-01T02:24:45.358 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-01T02:24:45.990 INFO:teuthology.orchestra.run.smithi062.stderr:Generating grub configuration file ... 2024-06-01T02:24:47.382 INFO:teuthology.orchestra.run.smithi062.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-01T02:24:47.413 INFO:teuthology.orchestra.run.smithi062.stderr:done 2024-06-01T02:24:47.415 DEBUG:teuthology.orchestra.run.smithi062:> mktemp 2024-06-01T02:24:47.433 INFO:teuthology.orchestra.run.smithi062.stdout:/tmp/tmp.GvohrpvTSj 2024-06-01T02:24:47.434 DEBUG:teuthology.orchestra.run.smithi062:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GvohrpvTSj 2024-06-01T02:24:47.500 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0666 /tmp/tmp.GvohrpvTSj 2024-06-01T02:24:47.639 DEBUG:teuthology.orchestra.remote:smithi062:/tmp/tmp.GvohrpvTSj is 6KB 2024-06-01T02:24:47.687 DEBUG:teuthology.orchestra.run.smithi062:> rm -fr /tmp/tmp.GvohrpvTSj 2024-06-01T02:24:47.701 DEBUG:teuthology.orchestra.run.smithi062:> sudo /bin/ls /boot/loader/entries || true 2024-06-01T02:24:47.772 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-01T02:24:47.772 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-01T02:24:47.772 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-01T02:24:47.772 INFO:teuthology.orchestra.run.smithi062.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-01T02:24:47.774 DEBUG:teuthology.orchestra.run.smithi062:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-01T02:24:47.968 DEBUG:teuthology.orchestra.run.smithi062:> sudo shutdown -r now 2024-06-01T02:25:08.982 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-01T02:25:08.983 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:25:08.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:25:17.978 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-01T02:25:17.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:25:17.979 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:25:27.326 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-06-01T02:25:36.334 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:25:36.335 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:25:36.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-06-01T02:25:39.394 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-06-01T02:25:45.546 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:25:45.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:25:48.606 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.62 2024-06-01T02:25:51.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:25:51.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:26:00.614 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:26:00.616 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:26:51.451 DEBUG:teuthology.orchestra.remote:timed out 2024-06-01T02:27:00.619 DEBUG:teuthology.orchestra.remote:timed out 2024-06-01T02:27:06.457 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:27:06.458 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:27:06.789 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-06-01T02:27:07.137 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:27:07.137 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-452.el9.x86_64"... 2024-06-01T02:27:07.137 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-06-01T02:27:07.154 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-452.el9.x86_64 2024-06-01T02:27:07.154 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-01T02:27:07.154 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-01T02:27:07.154 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-01T02:27:08.155 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-01T02:27:08.155 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-01T02:27:08.234 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-06-01T02:27:08.272 DEBUG:teuthology.parallel:result is None 2024-06-01T02:27:15.620 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:27:15.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:27:15.965 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-06-01T02:27:16.327 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:27:16.327 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-452.el9.x86_64"... 2024-06-01T02:27:16.327 DEBUG:teuthology.orchestra.run.smithi062:> uname -r 2024-06-01T02:27:16.384 INFO:teuthology.orchestra.run.smithi062.stdout:5.14.0-452.el9.x86_64 2024-06-01T02:27:16.384 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-01T02:27:16.384 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-01T02:27:16.384 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-01T02:27:17.385 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-01T02:27:17.386 DEBUG:teuthology.orchestra.run.smithi062:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-01T02:27:17.469 INFO:teuthology.orchestra.run.smithi062.stdout:ttyS1 2024-06-01T02:27:17.525 DEBUG:teuthology.parallel:result is None 2024-06-01T02:27:17.526 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-01T02:27:17.534 INFO:teuthology.task.internal:Creating test directory... 2024-06-01T02:27:17.534 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-01T02:27:17.539 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-01T02:27:17.558 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-01T02:27:17.565 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-01T02:27:17.571 INFO:teuthology.task.internal:Creating archive directory... 2024-06-01T02:27:17.572 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-01T02:27:17.598 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-01T02:27:17.631 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-01T02:27:17.638 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-01T02:27:17.638 DEBUG:teuthology.orchestra.run.smithi062:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-01T02:27:17.659 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-01T02:27:17.688 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T02:27:17.700 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T02:27:17.704 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T02:27:17.715 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-01T02:27:17.717 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-01T02:27:17.723 INFO:teuthology.task.internal:Configuring sudo... 2024-06-01T02:27:17.723 DEBUG:teuthology.orchestra.run.smithi062:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-01T02:27:17.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-01T02:27:17.783 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-01T02:27:17.792 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-01T02:27:17.792 DEBUG:teuthology.orchestra.run.smithi062:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-01T02:27:17.812 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-01T02:27:17.840 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T02:27:17.912 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T02:27:17.973 DEBUG:teuthology.orchestra.run.smithi062:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T02:27:18.015 DEBUG:teuthology.orchestra.run.smithi062:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T02:27:18.059 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:27:18.059 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-01T02:27:18.126 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T02:27:18.170 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-01T02:27:18.219 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T02:27:18.302 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-01T02:27:18.344 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:27:18.344 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-01T02:27:18.410 DEBUG:teuthology.orchestra.run.smithi062:> sudo service rsyslog restart 2024-06-01T02:27:18.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-06-01T02:27:18.450 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T02:27:18.484 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T02:27:18.984 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-01T02:27:18.993 INFO:teuthology.task.internal:Starting timer... 2024-06-01T02:27:18.993 INFO:teuthology.run_tasks:Running task pcp... 2024-06-01T02:27:19.003 INFO:teuthology.run_tasks:Running task selinux... 2024-06-01T02:27:19.012 DEBUG:teuthology.orchestra.run.smithi062:> sudo service auditd rotate 2024-06-01T02:27:19.088 INFO:teuthology.orchestra.run.smithi062.stdout:Rotating logs: 2024-06-01T02:27:19.090 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-06-01T02:27:19.140 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-06-01T02:27:19.143 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-01T02:27:19.143 DEBUG:teuthology.orchestra.run.smithi062:> /usr/sbin/getenforce 2024-06-01T02:27:19.164 INFO:teuthology.orchestra.run.smithi062.stdout:Permissive 2024-06-01T02:27:19.164 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-06-01T02:27:19.201 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-06-01T02:27:19.202 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi062.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-06-01T02:27:19.202 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-01T02:27:19.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:27:19.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-01T02:27:19.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:27:19.268 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-01T02:27:19.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo /usr/sbin/setenforce permissive 2024-06-01T02:27:19.308 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-06-01T02:27:19.344 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-01T02:27:19.354 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-01T02:27:19.357 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-01T02:27:19.481 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-01T02:27:19.493 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-01T02:27:19.494 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi062.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-01T02:31:45.745 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi062.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-06-01T02:31:45.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:31:45.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi062.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:31:45.827 DEBUG:teuthology.orchestra.run.smithi062:> true 2024-06-01T02:31:45.910 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi062.front.sepia.ceph.com' 2024-06-01T02:31:45.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:31:45.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-01T02:31:45.991 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-06-01T02:31:46.070 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-06-01T02:31:46.070 INFO:teuthology.run_tasks:Running task clock... 2024-06-01T02:31:46.084 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-01T02:31:46.084 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-01T02:31:46.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T02:31:46.087 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-01T02:31:46.088 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T02:31:46.116 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-01T02:31:46.131 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-01T02:31:46.140 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-01T02:31:46.155 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-01T02:31:46.156 INFO:teuthology.orchestra.run.smithi062.stderr:sudo: ntpd: command not found 2024-06-01T02:31:46.167 INFO:teuthology.orchestra.run.smithi062.stdout:506 Cannot talk to daemon 2024-06-01T02:31:46.180 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-06-01T02:31:46.181 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-01T02:31:46.192 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-06-01T02:31:46.196 INFO:teuthology.orchestra.run.smithi062.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-01T02:31:46.206 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-01T02:31:46.220 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-01T02:31:46.254 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-06-01T02:31:46.258 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T02:31:46.258 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-06-01T02:31:46.258 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.258 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.258 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.258 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.276 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-06-01T02:31:46.280 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T02:31:46.280 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-06-01T02:31:46.280 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.280 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.280 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.281 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:31:46.282 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-01T02:31:46.396 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-01T02:31:46.396 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-06-01T02:31:46.397 INFO:tasks.cephadm:Cluster fsid is 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:31:46.397 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-01T02:31:46.397 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.62', 'mon.c': '[v2:172.21.15.62:3301,v1:172.21.15.62:6790]', 'mon.b': '172.21.15.146'} 2024-06-01T02:31:46.397 INFO:tasks.cephadm:First mon is mon.a on smithi062 2024-06-01T02:31:46.397 INFO:tasks.cephadm:First mgr is y 2024-06-01T02:31:46.397 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-01T02:31:46.397 DEBUG:teuthology.orchestra.run.smithi062:> sudo hostname $(hostname -s) 2024-06-01T02:31:46.425 DEBUG:teuthology.orchestra.run.smithi146:> sudo hostname $(hostname -s) 2024-06-01T02:31:46.452 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-06-01T02:31:46.453 DEBUG:teuthology.orchestra.run.smithi062:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-01T02:31:46.693 INFO:teuthology.orchestra.run.smithi062.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Jun 1 02:31 /home/ubuntu/cephtest/cephadm 2024-06-01T02:31:46.695 DEBUG:teuthology.orchestra.run.smithi146:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-01T02:31:46.910 INFO:teuthology.orchestra.run.smithi146.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Jun 1 02:31 /home/ubuntu/cephtest/cephadm 2024-06-01T02:31:46.911 DEBUG:teuthology.orchestra.run.smithi062:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-01T02:31:46.929 DEBUG:teuthology.orchestra.run.smithi146:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-01T02:31:46.955 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-06-01T02:31:46.955 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-06-01T02:31:46.971 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-06-01T02:31:47.865 INFO:teuthology.orchestra.run.smithi146.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-06-01T02:31:47.886 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-06-01T02:32:04.216 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout: "repo_digests": [ 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-06-01T02:32:04.217 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:32:04.464 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-06-01T02:32:04.464 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-06-01T02:32:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-06-01T02:32:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "repo_digests": [ 2024-06-01T02:32:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-06-01T02:32:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-06-01T02:32:04.465 INFO:teuthology.orchestra.run.smithi146.stdout: ] 2024-06-01T02:32:04.465 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-06-01T02:32:04.483 DEBUG:teuthology.orchestra.run.smithi062:> sudo mkdir -p /etc/ceph 2024-06-01T02:32:04.513 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph 2024-06-01T02:32:04.541 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 777 /etc/ceph 2024-06-01T02:32:04.581 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /etc/ceph 2024-06-01T02:32:04.606 INFO:tasks.cephadm:Writing seed config... 2024-06-01T02:32:04.607 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-06-01T02:32:04.607 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-01T02:32:04.607 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-01T02:32:04.607 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-06-01T02:32:04.608 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-01T02:32:04.608 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-01T02:32:04.608 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-01T02:32:04.608 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-01T02:32:04.608 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-01T02:32:04.608 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:32:04.608 DEBUG:teuthology.orchestra.run.smithi062:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-01T02:32:04.638 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 176c65a6-1fbf-11ef-bc9b-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-06-01T02:32:04.639 DEBUG:teuthology.orchestra.run.smithi062:mon.a> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service 2024-06-01T02:32:04.682 DEBUG:teuthology.orchestra.run.smithi062:mgr.y> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service 2024-06-01T02:32:04.726 INFO:tasks.cephadm:Bootstrapping... 2024-06-01T02:32:04.727 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.62 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:32:04.929 INFO:teuthology.orchestra.run.smithi062.stderr:-------------------------------------------------------------------------------- 2024-06-01T02:32:04.929 INFO:teuthology.orchestra.run.smithi062.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '176c65a6-1fbf-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.62', '--skip-admin-label'] 2024-06-01T02:32:04.952 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 5.0.2 2024-06-01T02:32:04.955 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying podman|docker is present... 2024-06-01T02:32:04.977 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 5.0.2 2024-06-01T02:32:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying lvm2 is present... 2024-06-01T02:32:04.979 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying time synchronization is in place... 2024-06-01T02:32:04.987 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-06-01T02:32:04.993 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2024-06-01T02:32:05.000 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: enabled 2024-06-01T02:32:05.007 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: active 2024-06-01T02:32:05.008 INFO:teuthology.orchestra.run.smithi062.stderr:Unit chronyd.service is enabled and running 2024-06-01T02:32:05.008 INFO:teuthology.orchestra.run.smithi062.stderr:Repeating the final host check... 2024-06-01T02:32:05.029 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: 5.0.2 2024-06-01T02:32:05.031 INFO:teuthology.orchestra.run.smithi062.stderr:podman (/bin/podman) version 5.0.2 is present 2024-06-01T02:32:05.031 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl is present 2024-06-01T02:32:05.032 INFO:teuthology.orchestra.run.smithi062.stderr:lvcreate is present 2024-06-01T02:32:05.038 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-06-01T02:32:05.045 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2024-06-01T02:32:05.052 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: enabled 2024-06-01T02:32:05.058 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: active 2024-06-01T02:32:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:Unit chronyd.service is enabled and running 2024-06-01T02:32:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:Host looks OK 2024-06-01T02:32:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:Cluster fsid: 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:Acquiring lock 140149807182848 on /run/cephadm/176c65a6-1fbf-11ef-bc9b-c7b262605968.lock 2024-06-01T02:32:05.059 INFO:teuthology.orchestra.run.smithi062.stderr:Lock 140149807182848 acquired on /run/cephadm/176c65a6-1fbf-11ef-bc9b-c7b262605968.lock 2024-06-01T02:32:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying IP 172.21.15.62 port 3300 ... 2024-06-01T02:32:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying IP 172.21.15.62 port 6789 ... 2024-06-01T02:32:05.060 INFO:teuthology.orchestra.run.smithi062.stderr:Base mon IP is 172.21.15.62, final addrv is [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-06-01T02:32:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.62 metric 100 2024-06-01T02:32:05.063 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.62 metric 100 2024-06-01T02:32:05.065 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-06-01T02:32:05.065 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-01T02:32:05.065 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1796sec hoplimit 64 pref medium 2024-06-01T02:32:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-01T02:32:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-06-01T02:32:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-06-01T02:32:05.068 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-01T02:32:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febb:d1c3/64 scope link 2024-06-01T02:32:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-06-01T02:32:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:Mon IP `172.21.15.62` is in CIDR network `172.21.0.0/20` 2024-06-01T02:32:05.069 INFO:teuthology.orchestra.run.smithi062.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-01T02:32:05.070 INFO:teuthology.orchestra.run.smithi062.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-06-01T02:32:05.176 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Getting image source signatures 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-06-01T02:32:05.594 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-06-01T02:32:05.754 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: Writing manifest to image destination 2024-06-01T02:32:05.905 INFO:teuthology.orchestra.run.smithi062.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-06-01T02:32:06.282 INFO:teuthology.orchestra.run.smithi062.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-06-01T02:32:06.763 INFO:teuthology.orchestra.run.smithi062.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-06-01T02:32:06.763 INFO:teuthology.orchestra.run.smithi062.stderr:Extracting ceph user uid/gid from container image... 2024-06-01T02:32:07.188 INFO:teuthology.orchestra.run.smithi062.stderr:stat: 167 167 2024-06-01T02:32:07.577 INFO:teuthology.orchestra.run.smithi062.stderr:Creating initial keys... 2024-06-01T02:32:07.994 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-authtool: AQAniFpmbi6KNhAArVpNiDD0K3igV4B21mbabw== 2024-06-01T02:32:09.375 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-authtool: AQApiFpmZzUICxAAJZrG/t0tE1C+AD44RXsOrw== 2024-06-01T02:32:10.397 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph-authtool: AQAqiFpm6q1OExAAC7laYxoUTackELSEClSMPQ== 2024-06-01T02:32:10.829 INFO:teuthology.orchestra.run.smithi062.stderr:Creating initial monmap... 2024-06-01T02:32:11.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-01T02:32:11.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-06-01T02:32:11.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:11.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-01T02:32:11.810 INFO:teuthology.orchestra.run.smithi062.stderr:monmaptool for a [v2:172.21.15.62:3300,v1:172.21.15.62:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-01T02:32:11.810 INFO:teuthology.orchestra.run.smithi062.stderr:setting min_mon_release = octopus 2024-06-01T02:32:11.810 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: set fsid to 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:11.810 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-01T02:32:11.810 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:32:11.810 INFO:teuthology.orchestra.run.smithi062.stderr:Creating mon... 2024-06-01T02:32:12.848 INFO:teuthology.orchestra.run.smithi062.stderr:create mon.a on 2024-06-01T02:32:13.069 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-01T02:32:13.224 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968.target. 2024-06-01T02:32:13.224 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968.target. 2024-06-01T02:32:13.606 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to reset failed state of unit ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-06-01T02:32:13.614 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968.target.wants/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@.service. 2024-06-01T02:32:14.270 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:14 smithi062 podman[24732]: 2024-06-01 02:32:14.273526185 +0000 UTC m=+0.366833398 container start 2aad09728af204af49406264d9c32b7b7d697a2c164ed36e1e00eaaebf482bd1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_CLEAN=True, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-06-01T02:32:14.363 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2024-06-01T02:32:14.370 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2024-06-01T02:32:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2024-06-01T02:32:14.371 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to enable service . firewalld.service is not available 2024-06-01T02:32:14.372 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mon to start... 2024-06-01T02:32:14.372 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mon... 2024-06-01T02:32:14.584 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:14 smithi062 ceph-mon[24751]: mkfs 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:14.584 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:14 smithi062 ceph-mon[24751]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-01T02:32:14.584 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:14 smithi062 bash[24732]: 2aad09728af204af49406264d9c32b7b7d697a2c164ed36e1e00eaaebf482bd1 2024-06-01T02:32:14.584 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:14 smithi062 systemd[1]: Started Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:32:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: cluster: 2024-06-01T02:32:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: id: 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-06-01T02:32:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:14.892 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: services: 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.581928s) 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mgr: no daemons active 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: data: 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: pgs: 2024-06-01T02:32:14.893 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:15.473 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:15 smithi062 ceph-mon[24751]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-01T02:32:15.473 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:15 smithi062 ceph-mon[24751]: monmap e1: 1 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0]} 2024-06-01T02:32:15.473 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:15 smithi062 ceph-mon[24751]: fsmap 2024-06-01T02:32:15.473 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:15 smithi062 ceph-mon[24751]: osdmap e1: 0 total, 0 up, 0 in 2024-06-01T02:32:15.474 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:15 smithi062 ceph-mon[24751]: mgrmap e1: no daemons active 2024-06-01T02:32:15.474 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:15 smithi062 ceph-mon[24751]: from='client.? 172.21.15.62:0/3234874932' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-01T02:32:15.476 INFO:teuthology.orchestra.run.smithi062.stderr:mon is available 2024-06-01T02:32:15.476 INFO:teuthology.orchestra.run.smithi062.stderr:Assimilating anything we can from ceph.conf... 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [global] 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: fsid = 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.62:3300,v1:172.21.15.62:6789] 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-06-01T02:32:15.968 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [mgr] 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [osd] 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-06-01T02:32:15.969 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-06-01T02:32:16.461 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:16 smithi062 ceph-mon[24751]: from='client.? 172.21.15.62:0/3356242549' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-01T02:32:16.462 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:16 smithi062 ceph-mon[24751]: from='client.? 172.21.15.62:0/3356242549' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-01T02:32:16.464 INFO:teuthology.orchestra.run.smithi062.stderr:Generating new minimal ceph.conf... 2024-06-01T02:32:17.510 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:17 smithi062 ceph-mon[24751]: from='client.? 172.21.15.62:0/395472908' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:32:17.537 INFO:teuthology.orchestra.run.smithi062.stderr:Restarting the monitor... 2024-06-01T02:32:18.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:17 smithi062 systemd[1]: Stopping Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:32:18.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:17 smithi062 bash[24971]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon.a" found: no such container 2024-06-01T02:32:18.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a[24747]: 2024-06-01T02:32:17.681+0000 7f4acf674700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:32:18.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a[24747]: 2024-06-01T02:32:17.681+0000 7f4acf674700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-01T02:32:18.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:17 smithi062 podman[24982]: 2024-06-01 02:32:17.743525579 +0000 UTC m=+0.148580117 container died 2aad09728af204af49406264d9c32b7b7d697a2c164ed36e1e00eaaebf482bd1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, RELEASE=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-06-01T02:32:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 podman[24982]: 2024-06-01 02:32:18.089495665 +0000 UTC m=+0.494550206 container cleanup 2aad09728af204af49406264d9c32b7b7d697a2c164ed36e1e00eaaebf482bd1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., ceph=True, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container) 2024-06-01T02:32:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 bash[24982]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a 2024-06-01T02:32:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 podman[24994]: 2024-06-01 02:32:18.235479897 +0000 UTC m=+0.482668670 container remove 2aad09728af204af49406264d9c32b7b7d697a2c164ed36e1e00eaaebf482bd1 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, ceph=True, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-06-01T02:32:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 bash[25009]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon.a" found: no such container 2024-06-01T02:32:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-01T02:32:18.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 systemd[1]: Stopped Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:32:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 systemd[1]: Starting Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:32:18.481 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 podman[25076]: 2024-06-01 02:32:18.385997242 +0000 UTC m=+0.017942146 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-06-01T02:32:18.757 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 podman[25076]: 2024-06-01 02:32:18.482784636 +0000 UTC m=+0.114729523 container create 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, release=754, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=) 2024-06-01T02:32:18.757 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 podman[25076]: 2024-06-01 02:32:18.676761351 +0000 UTC m=+0.308706235 container init 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, ceph=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, version=8, vendor=Red Hat, Inc., name=centos-stream, maintainer=Guillaume Abrioux , RELEASE=HEAD, release=754, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=) 2024-06-01T02:32:18.757 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 podman[25076]: 2024-06-01 02:32:18.680052029 +0000 UTC m=+0.311996916 container start 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.buildah.version=1.19.8, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, version=8, io.k8s.display-name=CentOS Stream 8, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container) 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: pidfile_write: ignore empty --pid-file 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: load: jerasure load: lrc 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: RocksDB version: 6.15.5 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Compile date Apr 18 2022 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: DB SUMMARY 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: DB Session ID: IKNYXYBCI095QAUE4EZD 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: CURRENT file: CURRENT 2024-06-01T02:32:18.758 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82633 ; 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.error_if_exists: 0 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.create_if_missing: 0 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.paranoid_checks: 1 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.env: 0x56044e8c6860 2024-06-01T02:32:18.759 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.fs: Posix File System 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.info_log: 0x560450345dc0 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.statistics: (nil) 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.use_fsync: 0 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_log_file_size: 0 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T02:32:18.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.allow_fallocate: 1 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.use_direct_reads: 0 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.db_log_dir: 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T02:32:18.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.write_buffer_manager: 0x5604504362a0 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T02:32:18.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.unordered_write: 0 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.row_cache: None 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.wal_filter: None 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T02:32:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T02:32:18.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.preserve_deletes: 0 2024-06-01T02:32:18.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.two_write_queues: 0 2024-06-01T02:32:18.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T02:32:18.765 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.atomic_flush: 0 2024-06-01T02:32:18.765 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T02:32:18.765 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.log_readahead_size: 0 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_background_jobs: 2 2024-06-01T02:32:18.766 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_background_compactions: -1 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_subcompactions: 1 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_open_files: -1 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T02:32:18.767 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_background_flushes: -1 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Compression algorithms supported: 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kZSTD supported: 0 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kXpressCompression supported: 0 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kLZ4Compression supported: 1 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kBZip2Compression supported: 0 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kZlibCompression supported: 1 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: kSnappyCompression supported: 1 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-06-01T02:32:18.768 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.merge_operator: 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_filter: None 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x560450311d00) 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_type: 0 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: data_block_index_type: 0 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_shortening: 1 2024-06-01T02:32:18.769 INFO:journalctl@ceph.mon.a.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: hash_index_allow_collision: 1 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: checksum: 1 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: no_block_cache: 0 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache: 0x56045037c170 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_options: 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: capacity : 536870912 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: num_shard_bits : 4 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: strict_capacity_limit : 0 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_compressed: (nil) 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: persistent_cache: (nil) 2024-06-01T02:32:18.770 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_size: 4096 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_size_deviation: 10 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_restart_interval: 16 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_block_restart_interval: 1 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: metadata_block_size: 4096 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: partition_filters: 0 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: use_delta_encoding: 1 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: whole_key_filtering: 1 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: verify_compression: 0 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: format_version: 4 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: enable_index_compression: 1 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_align: 0 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T02:32:18.771 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression: NoCompression 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.num_levels: 7 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T02:32:18.772 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T02:32:18.773 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.arena_block_size: 4194304 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T02:32:18.774 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.table_properties_collectors: 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.inplace_update_support: 0 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T02:32:18.775 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.bloom_locality: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.max_successive_merges: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.ttl: 2592000 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.enable_blob_files: false 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.min_blob_size: 0 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T02:32:18.776 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209138706183, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209138706981, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79183, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77446, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71755, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1717209138, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9795e1c8-2d14-499b-961e-18b33168d883", "db_session_id": "IKNYXYBCI095QAUE4EZD"}} 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209138707314, "job": 1, "event": "recovery_finished"} 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x560450362a80 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: DB pointer 0x560450372000 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-06-01T02:32:18.777 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** DB Stats ** 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:32:18.778 INFO:journalctl@ceph.mon.a.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:32:18.779 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative compaction: 0.00 GB write, 24.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval compaction: 0.00 GB write, 24.16 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:32:18.780 INFO:journalctl@ceph.mon.a.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 141.1 0.00 0.00 1 0.001 0 0 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative compaction: 0.00 GB write, 23.92 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: starting mon.a rank 0 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:18.781 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???) e1 preinit fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).mds e1 new map 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).mds e1 print_map 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout: e1 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout: legacy client fscid: -1 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout: No filesystems configured 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T02:32:18.782 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-01T02:32:18.783 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 bash[25076]: 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 2024-06-01T02:32:18.783 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 systemd[1]: Started Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:32:18.783 INFO:teuthology.orchestra.run.smithi062.stderr:Setting mon public_network to 172.21.0.0/20 2024-06-01T02:32:19.010 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mon.a is new leader, mons a in quorum (ranks 0) 2024-06-01T02:32:19.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: monmap e1: 1 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0]} 2024-06-01T02:32:19.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: fsmap 2024-06-01T02:32:19.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: osdmap e1: 0 total, 0 up, 0 in 2024-06-01T02:32:19.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:18 smithi062 ceph-mon[25095]: mgrmap e1: no daemons active 2024-06-01T02:32:19.783 INFO:teuthology.orchestra.run.smithi062.stderr:Wrote config to /etc/ceph/ceph.conf 2024-06-01T02:32:19.783 INFO:teuthology.orchestra.run.smithi062.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:32:19.783 INFO:teuthology.orchestra.run.smithi062.stderr:Creating mgr... 2024-06-01T02:32:19.784 INFO:teuthology.orchestra.run.smithi062.stderr:Verifying port 9283 ... 2024-06-01T02:32:19.963 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Failed to reset failed state of unit ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service: Unit ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service not loaded. 2024-06-01T02:32:19.971 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: Created symlink /etc/systemd/system/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968.target.wants/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@.service. 2024-06-01T02:32:20.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 systemd[1]: Starting Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:32:20.575 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:20 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3228932183' entity='client.admin' 2024-06-01T02:32:20.576 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 podman[25306]: 2024-06-01 02:32:20.265257926 +0000 UTC m=+0.019689967 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-06-01T02:32:20.576 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 podman[25306]: 2024-06-01 02:32:20.383822218 +0000 UTC m=+0.138254231 container create 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, release=754, io.buildah.version=1.19.8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=) 2024-06-01T02:32:20.576 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 podman[25306]: 2024-06-01 02:32:20.578222272 +0000 UTC m=+0.332654297 container init 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vendor=Red Hat, Inc., ceph=True, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream) 2024-06-01T02:32:20.697 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2024-06-01T02:32:20.704 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2024-06-01T02:32:20.705 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2024-06-01T02:32:20.705 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to enable service . firewalld.service is not available 2024-06-01T02:32:20.713 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2024-06-01T02:32:20.721 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2024-06-01T02:32:20.722 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2024-06-01T02:32:20.722 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-01T02:32:20.722 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr to start... 2024-06-01T02:32:20.722 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr... 2024-06-01T02:32:20.849 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 podman[25306]: 2024-06-01 02:32:20.581807588 +0000 UTC m=+0.336239609 container start 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-type=git, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, version=8, release=754, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, ceph=True, architecture=x86_64, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-06-01T02:32:20.849 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 bash[25306]: 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 2024-06-01T02:32:20.849 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 systemd[1]: Started Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:32:20.849 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:20.729+0000 7fc664a2b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:32:21.144 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:20.867+0000 7fc664a2b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:32:21.144 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:20.971+0000 7fc664a2b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:32:21.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:21.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:32:21.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "176c65a6-1fbf-11ef-bc9b-c7b262605968", 2024-06-01T02:32:21.245 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "a" 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2024-06-01T02:32:21.246 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-01T02:32:21.247 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-01T02:32:21.250 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-01T02:32:21.250 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:21.251 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": false, 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "nfs", 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2024-06-01T02:32:21.252 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:21.253 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2024-06-01T02:32:14.313089+0000", 2024-06-01T02:32:21.253 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:21.253 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:21.253 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-01T02:32:21.253 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:32:21.510 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:21 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2411674440' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T02:32:21.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:21.304+0000 7fc664a2b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:32:21.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:21.390+0000 7fc664a2b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:32:21.837 INFO:teuthology.orchestra.run.smithi062.stderr:mgr not available, waiting (1/15)... 2024-06-01T02:32:22.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:22 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:22.585+0000 7fc664a2b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:32:22.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:22 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:22.754+0000 7fc664a2b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:32:22.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:22 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:22.838+0000 7fc664a2b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:32:23.354 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:23.167+0000 7fc664a2b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:32:23.354 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:23.260+0000 7fc664a2b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:32:23.354 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:23.356+0000 7fc664a2b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:32:23.760 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:23.605+0000 7fc664a2b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:32:24.260 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:24.073+0000 7fc664a2b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "176c65a6-1fbf-11ef-bc9b-c7b262605968", 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2024-06-01T02:32:24.354 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "a" 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-01T02:32:24.355 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-01T02:32:24.356 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": false, 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "nfs", 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2024-06-01T02:32:24.357 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2024-06-01T02:32:14.313089+0000", 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-01T02:32:24.358 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:32:24.946 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:24 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2844426846' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T02:32:24.946 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:24.588+0000 7fc664a2b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:32:24.946 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:24.685+0000 7fc664a2b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:32:24.948 INFO:teuthology.orchestra.run.smithi062.stderr:mgr not available, waiting (2/15)... 2024-06-01T02:32:25.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:24.993+0000 7fc664a2b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:32:25.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:25.086+0000 7fc664a2b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:32:25.760 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:25.504+0000 7fc664a2b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:32:25.760 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:25.642+0000 7fc664a2b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:32:25.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:25.737+0000 7fc664a2b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:32:26.260 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:25.855+0000 7fc664a2b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:32:26.260 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:25.947+0000 7fc664a2b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:32:26.893 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:26.518+0000 7fc664a2b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:32:27.206 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:26.895+0000 7fc664a2b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: Activating manager daemon y 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: mgrmap e2: y(active, starting, since 0.00275771s) 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: Manager daemon y is now available 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' 2024-06-01T02:32:27.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:26 smithi062 ceph-mon[25095]: from='mgr.14100 172.21.15.62:0/1885642394' entity='mgr.y' 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "176c65a6-1fbf-11ef-bc9b-c7b262605968", 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2024-06-01T02:32:27.384 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "a" 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-01T02:32:27.385 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2024-06-01T02:32:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-01T02:32:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-01T02:32:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-01T02:32:27.386 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-01T02:32:27.389 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": false, 2024-06-01T02:32:27.390 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "nfs", 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2024-06-01T02:32:14.313089+0000", 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-01T02:32:27.391 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:32:28.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:27 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3812645792' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T02:32:28.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:27 smithi062 ceph-mon[25095]: mgrmap e3: y(active, since 1.00678s) 2024-06-01T02:32:31.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:30 smithi062 ceph-mon[25095]: mgrmap e4: y(active, since 2s) 2024-06-01T02:32:31.979 INFO:teuthology.orchestra.run.smithi062.stderr:mgr not available, waiting (3/15)... 2024-06-01T02:32:37.506 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsid": "176c65a6-1fbf-11ef-bc9b-c7b262605968", 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "health": { 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "checks": {}, 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mutes": [] 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum": [ 2024-06-01T02:32:37.507 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 0 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_names": [ 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "a" 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "quorum_age": 18, 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "monmap": { 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_mons": 1 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osdmap": { 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_osds": 0, 2024-06-01T02:32:37.508 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgmap": { 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_pools": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_objects": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-06-01T02:32:37.509 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "bytes_total": 0 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "fsmap": { 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "by_rank": [], 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "up:standby": 0 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap": { 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": true, 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modules": [ 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "iostat", 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "nfs", 2024-06-01T02:32:37.510 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "restful" 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ], 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "servicemap": { 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 1, 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "modified": "2024-06-01T02:32:14.313089+0000", 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "services": {} 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: }, 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "progress_events": {} 2024-06-01T02:32:37.511 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:32:37.690 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:37 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/642087797' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-01T02:32:38.044 INFO:teuthology.orchestra.run.smithi062.stderr:mgr is available 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [global] 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: fsid = 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:38.645 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [mgr] 2024-06-01T02:32:38.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-06-01T02:32:38.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 2024-06-01T02:32:38.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: [osd] 2024-06-01T02:32:38.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-06-01T02:32:38.646 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-06-01T02:32:38.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:38 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3840912301' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-01T02:32:38.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:38 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3840912301' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-01T02:32:39.224 INFO:teuthology.orchestra.run.smithi062.stderr:Enabling cephadm module... 2024-06-01T02:32:40.196 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:39 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1605918519' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-01T02:32:40.197 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:39 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setuser ceph since I am not root 2024-06-01T02:32:40.197 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:39 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setgroup ceph since I am not root 2024-06-01T02:32:40.197 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:40.067+0000 7f18f1f20000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:32:40.510 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:40.204+0000 7f18f1f20000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:32:40.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:40.307+0000 7f18f1f20000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:32:40.899 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:40.626+0000 7f18f1f20000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:32:40.899 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:40.711+0000 7f18f1f20000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:32:41.243 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1605918519' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-01T02:32:41.243 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:40 smithi062 ceph-mon[25095]: mgrmap e5: y(active, since 13s) 2024-06-01T02:32:41.497 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:32:41.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 5, 2024-06-01T02:32:41.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": true, 2024-06-01T02:32:41.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "active_name": "y", 2024-06-01T02:32:41.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standby": 0 2024-06-01T02:32:41.498 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:32:42.100 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for the mgr to restart... 2024-06-01T02:32:42.100 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr epoch 5... 2024-06-01T02:32:42.179 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:41 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1888452860' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-01T02:32:42.180 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:41 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:41.914+0000 7f18f1f20000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:32:42.180 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:42.086+0000 7f18f1f20000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:32:42.180 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:42.181+0000 7f18f1f20000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:32:42.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:42.512+0000 7f18f1f20000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:32:42.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:42.606+0000 7f18f1f20000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:32:42.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:42.702+0000 7f18f1f20000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:32:43.260 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:42.951+0000 7f18f1f20000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:32:43.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:43.411+0000 7f18f1f20000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:32:44.260 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:43.913+0000 7f18f1f20000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:32:44.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:44.009+0000 7f18f1f20000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:32:44.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:44.314+0000 7f18f1f20000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:32:44.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:44.407+0000 7f18f1f20000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:32:45.170 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:44.823+0000 7f18f1f20000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:32:45.170 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:44.960+0000 7f18f1f20000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:32:45.171 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:45.055+0000 7f18f1f20000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:32:45.171 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:45.172+0000 7f18f1f20000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:32:45.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:45.263+0000 7f18f1f20000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:32:46.203 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:45.833+0000 7f18f1f20000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:32:46.510 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:46 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:32:46.205+0000 7f18f1f20000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:32:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:32:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-06-01T02:32:47.268 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "initialized": true 2024-06-01T02:32:47.269 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:32:47.499 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:32:47] ENGINE Bus STARTING 2024-06-01T02:32:47.499 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:32:47] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:32:47.499 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: Active manager daemon y restarted 2024-06-01T02:32:47.500 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: Activating manager daemon y 2024-06-01T02:32:47.500 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: osdmap e2: 0 total, 0 up, 0 in 2024-06-01T02:32:47.500 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: mgrmap e6: y(active, starting, since 0.0524787s) 2024-06-01T02:32:47.500 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:32:47.500 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:32:47.500 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: Manager daemon y is now available 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:32:47.501 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:32:47.502 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:32:47.502 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:32:47.502 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:32:47.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:32:47] ENGINE Bus STARTED 2024-06-01T02:32:47.850 INFO:teuthology.orchestra.run.smithi062.stderr:mgr epoch 5 is available 2024-06-01T02:32:47.851 INFO:teuthology.orchestra.run.smithi062.stderr:Setting orchestrator backend to cephadm... 2024-06-01T02:32:48.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:48 smithi062 ceph-mon[25095]: mgrmap e7: y(active, since 1.05483s) 2024-06-01T02:32:48.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:48 smithi062 ceph-mon[25095]: [01/Jun/2024:02:32:47] ENGINE Bus STARTING 2024-06-01T02:32:48.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:48 smithi062 ceph-mon[25095]: [01/Jun/2024:02:32:47] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:32:48.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:48 smithi062 ceph-mon[25095]: [01/Jun/2024:02:32:47] ENGINE Bus STARTED 2024-06-01T02:32:48.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:48 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:48.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:48 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:32:49.608 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: value unchanged 2024-06-01T02:32:49.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:49 smithi062 ceph-mon[25095]: mgrmap e8: y(active, since 2s) 2024-06-01T02:32:49.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:49 smithi062 ceph-mon[25095]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:49.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:49 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:49.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:49 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:32:50.131 INFO:teuthology.orchestra.run.smithi062.stderr:Generating ssh key... 2024-06-01T02:32:50.760 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:50 smithi062 ceph-mon[25095]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-mon[25095]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-mon[25095]: Generating ssh key... 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: Generating public/private rsa key pair. 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: Your identification has been saved in /tmp/tmpjp8qd4mt/key. 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: Your public key has been saved in /tmp/tmpjp8qd4mt/key.pub. 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: The key fingerprint is: 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: SHA256:PT8vOAtNcIOMrFZCjUody7hlDcSoWCUCFwj/TYmvZ5M ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: The key's randomart image is: 2024-06-01T02:32:51.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: +---[RSA 3072]----+ 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: |*.+B*+ | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: |.++=+*.+ . | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: |.+o.B B + o | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: |o .= B + . | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: | . + o S + | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: | . . . o o | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: | . E . ..o | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: | o . .o .o | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: | .o .. | 2024-06-01T02:32:51.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:32:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: +----[SHA256]-----+ 2024-06-01T02:32:52.696 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR0yr7/vIWI2GD3n6/AIuKbIUhEGTAgyZ68++pnSAEuHUt57RI2SVmRP7ialBA1EThGSaKZZFwkjcBctKaAGfLDX/P3B8Cf3ACBAcc7IS2rP0hg8JI1QgMsI9sK3kxwNLMcrccGPMvjm0nH/vsFK06cW1q+BLmG+luVgeXVoC0PX3HS7g2leqDopvBQzyRQMvwLs/MK03fpa6p8bhHKM+0WtcWDCwCzmL1pvd5/usBOoCvJMrx0AfRUTzqk9jbQR3EgVQ+JskOC1q+mUj8cuJFDhVloc+0mIN/gLZpn9w6kTg32iHD+7CHnamsS23Hs1p6VBKYDlWdL8sQaKkC5sxyaIGt87kmx4HJ6wOsPKbqnz8PN/ocwmIwTOerQ3Y6PqlzH2IVWdNs+lHlyfWFXEukimLMUZ34RBrNtuuK46OYVIfJQ6K/+K5hz8l0FGjYOe2A4vZ3fjMMxz2eYboG2Takd4SMmY7tTij5AGZkp/zrOKBcCnaPF+Cs1oT0pMYStTk= ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:32:52.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:52 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:52.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:52 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-01T02:32:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:Adding key to root@localhost authorized_keys... 2024-06-01T02:32:53.190 INFO:teuthology.orchestra.run.smithi062.stderr:Adding host smithi062... 2024-06-01T02:32:53.830 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:53 smithi062 ceph-mon[25095]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:54.685 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Added host 'smithi062' with addr '172.21.15.62' 2024-06-01T02:32:54.990 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:54 smithi062 ceph-mon[25095]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi062", "addr": "172.21.15.62", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:54.990 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:54 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:54.990 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:54 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:32:55.276 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying unmanaged mon service... 2024-06-01T02:32:55.912 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:55 smithi062 ceph-mon[25095]: Deploying cephadm binary to smithi062 2024-06-01T02:32:55.912 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:55 smithi062 ceph-mon[25095]: Added host smithi062 2024-06-01T02:32:55.912 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled mon update... 2024-06-01T02:32:56.476 INFO:teuthology.orchestra.run.smithi062.stderr:Deploying unmanaged mgr service... 2024-06-01T02:32:57.202 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:56 smithi062 ceph-mon[25095]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:57.202 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:56 smithi062 ceph-mon[25095]: Saving service mon spec with placement count:5 2024-06-01T02:32:57.202 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:56 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:57.319 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-06-01T02:32:58.670 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:58 smithi062 ceph-mon[25095]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:32:58.671 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:58 smithi062 ceph-mon[25095]: Saving service mgr spec with placement count:2 2024-06-01T02:32:58.671 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:58 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:32:58.671 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:58 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:33:00.010 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:59 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:33:00.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:32:59 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3253446858' entity='client.admin' 2024-06-01T02:33:01.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:00 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:33:01.554 INFO:teuthology.orchestra.run.smithi062.stderr:Enabling the dashboard module... 2024-06-01T02:33:02.163 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:01 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1285931967' entity='client.admin' 2024-06-01T02:33:02.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:01 smithi062 ceph-mon[25095]: from='mgr.14120 172.21.15.62:0/1908135588' entity='mgr.y' 2024-06-01T02:33:03.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:02 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3125998979' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-01T02:33:03.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setuser ceph since I am not root 2024-06-01T02:33:03.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setgroup ceph since I am not root 2024-06-01T02:33:03.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:03.143+0000 7fe5ed964000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:33:03.672 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:03.282+0000 7fe5ed964000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:33:03.672 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:03.385+0000 7fe5ed964000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:33:03.942 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:03.708+0000 7fe5ed964000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:33:03.942 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:03.795+0000 7fe5ed964000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:33:04.260 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3125998979' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-01T02:33:04.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:03 smithi062 ceph-mon[25095]: mgrmap e9: y(active, since 16s) 2024-06-01T02:33:04.445 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:33:04.445 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "epoch": 9, 2024-06-01T02:33:04.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "available": true, 2024-06-01T02:33:04.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "active_name": "y", 2024-06-01T02:33:04.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "num_standby": 0 2024-06-01T02:33:04.446 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:33:05.095 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:04 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/548701953' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-01T02:33:05.096 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:04.992+0000 7fe5ed964000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:33:05.098 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for the mgr to restart... 2024-06-01T02:33:05.098 INFO:teuthology.orchestra.run.smithi062.stderr:Waiting for mgr epoch 9... 2024-06-01T02:33:05.365 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:05.171+0000 7fe5ed964000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:33:05.365 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:05.257+0000 7fe5ed964000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:33:05.680 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:05.588+0000 7fe5ed964000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:33:06.011 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:05.682+0000 7fe5ed964000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:33:06.011 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:05.777+0000 7fe5ed964000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:33:06.483 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:06.026+0000 7fe5ed964000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:33:06.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:06.485+0000 7fe5ed964000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:33:07.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:06.988+0000 7fe5ed964000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:33:07.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:07.084+0000 7fe5ed964000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:33:07.681 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:07.392+0000 7fe5ed964000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:33:07.681 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:07.487+0000 7fe5ed964000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:33:08.010 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:07.915+0000 7fe5ed964000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:33:08.359 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:08.054+0000 7fe5ed964000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:33:08.360 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:08.150+0000 7fe5ed964000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:33:08.360 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:08.268+0000 7fe5ed964000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:33:08.360 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:08.361+0000 7fe5ed964000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:33:09.261 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:08.931+0000 7fe5ed964000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:33:09.717 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:09 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:33:09.303+0000 7fe5ed964000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:33:10.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:09 smithi062 ceph-mon[25095]: Active manager daemon y restarted 2024-06-01T02:33:10.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:09 smithi062 ceph-mon[25095]: Activating manager daemon y 2024-06-01T02:33:10.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:09 smithi062 ceph-mon[25095]: osdmap e3: 0 total, 0 up, 0 in 2024-06-01T02:33:10.403 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:33:10] ENGINE Bus STARTING 2024-06-01T02:33:10.403 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:33:10] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:33:10.403 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:33:10] ENGINE Bus STARTED 2024-06-01T02:33:10.736 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: { 2024-06-01T02:33:10.736 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-06-01T02:33:10.736 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: "initialized": true 2024-06-01T02:33:10.737 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: } 2024-06-01T02:33:10.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: mgrmap e10: y(active, starting, since 0.41934s) 2024-06-01T02:33:10.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:33:10.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:33:10.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:33:10.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:33:10.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:33:10.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: Manager daemon y is now available 2024-06-01T02:33:10.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:10.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:10.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:33:10.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:33:10.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:11.283 INFO:teuthology.orchestra.run.smithi062.stderr:mgr epoch 9 is available 2024-06-01T02:33:11.284 INFO:teuthology.orchestra.run.smithi062.stderr:Generating a dashboard self-signed certificate... 2024-06-01T02:33:11.805 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: [01/Jun/2024:02:33:10] ENGINE Bus STARTING 2024-06-01T02:33:11.805 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: [01/Jun/2024:02:33:10] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:33:11.805 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: [01/Jun/2024:02:33:10] ENGINE Bus STARTED 2024-06-01T02:33:11.805 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-01T02:33:11.805 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: mgrmap e11: y(active, since 1.42461s) 2024-06-01T02:33:11.805 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-01T02:33:11.806 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:11 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:12.065 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: Self-signed certificate created 2024-06-01T02:33:13.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:13.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:12 smithi062 ceph-mon[25095]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:33:13.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:12 smithi062 ceph-mon[25095]: mgrmap e12: y(active, since 2s) 2024-06-01T02:33:13.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:13.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:13.403 INFO:teuthology.orchestra.run.smithi062.stderr:Creating initial admin user... 2024-06-01T02:33:14.711 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$8k1zn3IAZnQPn98mKvxNv.N0WrGwtVagn0gBG.6NIpSZh5c2dWJei", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717209194, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-01T02:33:15.312 INFO:teuthology.orchestra.run.smithi062.stderr:Fetching dashboard port number... 2024-06-01T02:33:15.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:15 smithi062 ceph-mon[25095]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:33:15.946 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:15.946 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: 8443 2024-06-01T02:33:16.554 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: disabled 2024-06-01T02:33:16.562 INFO:teuthology.orchestra.run.smithi062.stderr:systemctl: inactive 2024-06-01T02:33:16.563 INFO:teuthology.orchestra.run.smithi062.stderr:firewalld.service is not enabled 2024-06-01T02:33:16.563 INFO:teuthology.orchestra.run.smithi062.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-01T02:33:16.565 INFO:teuthology.orchestra.run.smithi062.stderr:Ceph Dashboard is now available at: 2024-06-01T02:33:16.565 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:16.565 INFO:teuthology.orchestra.run.smithi062.stderr: URL: https://smithi062.front.sepia.ceph.com:8443/ 2024-06-01T02:33:16.565 INFO:teuthology.orchestra.run.smithi062.stderr: User: admin 2024-06-01T02:33:16.566 INFO:teuthology.orchestra.run.smithi062.stderr: Password: 9x9ftgcgti 2024-06-01T02:33:16.566 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:16.566 INFO:teuthology.orchestra.run.smithi062.stderr:Enabling autotune for osd_memory_target 2024-06-01T02:33:16.820 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:16 smithi062 ceph-mon[25095]: mgrmap e13: y(active, since 6s) 2024-06-01T02:33:16.820 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:16 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3186882940' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-01T02:33:18.574 INFO:teuthology.orchestra.run.smithi062.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-06-01T02:33:19.183 INFO:teuthology.orchestra.run.smithi062.stderr:You can access the Ceph CLI with: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr:Please consider enabling telemetry to help improve Ceph: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: ceph telemetry on 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr:For more information see: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:19.184 INFO:teuthology.orchestra.run.smithi062.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-01T02:33:19.185 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:33:19.185 INFO:teuthology.orchestra.run.smithi062.stderr:Bootstrap complete. 2024-06-01T02:33:19.229 INFO:tasks.cephadm:Fetching config... 2024-06-01T02:33:19.229 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:33:19.229 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-01T02:33:19.247 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-01T02:33:19.248 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:33:19.248 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-01T02:33:19.305 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-01T02:33:19.305 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:33:19.305 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-06-01T02:33:19.373 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-01T02:33:19.373 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:33:19.373 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-01T02:33:19.430 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-01T02:33:19.431 DEBUG:teuthology.orchestra.run.smithi062:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR0yr7/vIWI2GD3n6/AIuKbIUhEGTAgyZ68++pnSAEuHUt57RI2SVmRP7ialBA1EThGSaKZZFwkjcBctKaAGfLDX/P3B8Cf3ACBAcc7IS2rP0hg8JI1QgMsI9sK3kxwNLMcrccGPMvjm0nH/vsFK06cW1q+BLmG+luVgeXVoC0PX3HS7g2leqDopvBQzyRQMvwLs/MK03fpa6p8bhHKM+0WtcWDCwCzmL1pvd5/usBOoCvJMrx0AfRUTzqk9jbQR3EgVQ+JskOC1q+mUj8cuJFDhVloc+0mIN/gLZpn9w6kTg32iHD+7CHnamsS23Hs1p6VBKYDlWdL8sQaKkC5sxyaIGt87kmx4HJ6wOsPKbqnz8PN/ocwmIwTOerQ3Y6PqlzH2IVWdNs+lHlyfWFXEukimLMUZ34RBrNtuuK46OYVIfJQ6K/+K5hz8l0FGjYOe2A4vZ3fjMMxz2eYboG2Takd4SMmY7tTij5AGZkp/zrOKBcCnaPF+Cs1oT0pMYStTk= ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-01T02:33:19.509 INFO:teuthology.orchestra.run.smithi062.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR0yr7/vIWI2GD3n6/AIuKbIUhEGTAgyZ68++pnSAEuHUt57RI2SVmRP7ialBA1EThGSaKZZFwkjcBctKaAGfLDX/P3B8Cf3ACBAcc7IS2rP0hg8JI1QgMsI9sK3kxwNLMcrccGPMvjm0nH/vsFK06cW1q+BLmG+luVgeXVoC0PX3HS7g2leqDopvBQzyRQMvwLs/MK03fpa6p8bhHKM+0WtcWDCwCzmL1pvd5/usBOoCvJMrx0AfRUTzqk9jbQR3EgVQ+JskOC1q+mUj8cuJFDhVloc+0mIN/gLZpn9w6kTg32iHD+7CHnamsS23Hs1p6VBKYDlWdL8sQaKkC5sxyaIGt87kmx4HJ6wOsPKbqnz8PN/ocwmIwTOerQ3Y6PqlzH2IVWdNs+lHlyfWFXEukimLMUZ34RBrNtuuK46OYVIfJQ6K/+K5hz8l0FGjYOe2A4vZ3fjMMxz2eYboG2Takd4SMmY7tTij5AGZkp/zrOKBcCnaPF+Cs1oT0pMYStTk= ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:33:19.520 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR0yr7/vIWI2GD3n6/AIuKbIUhEGTAgyZ68++pnSAEuHUt57RI2SVmRP7ialBA1EThGSaKZZFwkjcBctKaAGfLDX/P3B8Cf3ACBAcc7IS2rP0hg8JI1QgMsI9sK3kxwNLMcrccGPMvjm0nH/vsFK06cW1q+BLmG+luVgeXVoC0PX3HS7g2leqDopvBQzyRQMvwLs/MK03fpa6p8bhHKM+0WtcWDCwCzmL1pvd5/usBOoCvJMrx0AfRUTzqk9jbQR3EgVQ+JskOC1q+mUj8cuJFDhVloc+0mIN/gLZpn9w6kTg32iHD+7CHnamsS23Hs1p6VBKYDlWdL8sQaKkC5sxyaIGt87kmx4HJ6wOsPKbqnz8PN/ocwmIwTOerQ3Y6PqlzH2IVWdNs+lHlyfWFXEukimLMUZ34RBrNtuuK46OYVIfJQ6K/+K5hz8l0FGjYOe2A4vZ3fjMMxz2eYboG2Takd4SMmY7tTij5AGZkp/zrOKBcCnaPF+Cs1oT0pMYStTk= ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-01T02:33:19.563 INFO:teuthology.orchestra.run.smithi146.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR0yr7/vIWI2GD3n6/AIuKbIUhEGTAgyZ68++pnSAEuHUt57RI2SVmRP7ialBA1EThGSaKZZFwkjcBctKaAGfLDX/P3B8Cf3ACBAcc7IS2rP0hg8JI1QgMsI9sK3kxwNLMcrccGPMvjm0nH/vsFK06cW1q+BLmG+luVgeXVoC0PX3HS7g2leqDopvBQzyRQMvwLs/MK03fpa6p8bhHKM+0WtcWDCwCzmL1pvd5/usBOoCvJMrx0AfRUTzqk9jbQR3EgVQ+JskOC1q+mUj8cuJFDhVloc+0mIN/gLZpn9w6kTg32iHD+7CHnamsS23Hs1p6VBKYDlWdL8sQaKkC5sxyaIGt87kmx4HJ6wOsPKbqnz8PN/ocwmIwTOerQ3Y6PqlzH2IVWdNs+lHlyfWFXEukimLMUZ34RBrNtuuK46OYVIfJQ6K/+K5hz8l0FGjYOe2A4vZ3fjMMxz2eYboG2Takd4SMmY7tTij5AGZkp/zrOKBcCnaPF+Cs1oT0pMYStTk= ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:33:19.576 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-01T02:33:19.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:19 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2013657971' entity='client.admin' 2024-06-01T02:33:22.259 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:22 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:22.259 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:22 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi062", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:33:22.259 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:22 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:22.259 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:22 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:22.259 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:22 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:23.503 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-01T02:33:23.504 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-01T02:33:23.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:23 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4256980666' entity='client.admin' 2024-06-01T02:33:23.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:23 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:23.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:23 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:25.938 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi146 2024-06-01T02:33:25.938 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:33:25.938 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.conf 2024-06-01T02:33:25.957 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:33:25.958 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:33:26.016 INFO:tasks.cephadm:Adding host smithi146 to orchestrator... 2024-06-01T02:33:26.016 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch host add smithi146 2024-06-01T02:33:26.566 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:26.567 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:28.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:28 smithi062 ceph-mon[25095]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi146", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:33:28.864 INFO:teuthology.orchestra.run.smithi062.stdout:Added host 'smithi146' with addr '172.21.15.146' 2024-06-01T02:33:29.451 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-01T02:33:30.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: Deploying cephadm binary to smithi146 2024-06-01T02:33:30.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:30.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: Added host smithi146 2024-06-01T02:33:30.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:30.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:30.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:30.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:30.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:30.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:30 smithi062 ceph-mon[25095]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:30.945 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:30 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:31.334 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:33:31.335 INFO:teuthology.orchestra.run.smithi062.stdout:[{"addr": "172.21.15.62", "hostname": "smithi062", "labels": [], "status": ""}, {"addr": "172.21.15.146", "hostname": "smithi146", "labels": [], "status": ""}] 2024-06-01T02:33:31.921 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-01T02:33:31.922 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-01T02:33:33.120 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:32 smithi062 ceph-mon[25095]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:33:33.120 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:32 smithi062 ceph-mon[25095]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:34.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:34 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:34.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:34 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:34.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:34 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi146", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:33:34.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:34 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:34.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:34 smithi062 ceph-mon[25095]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:34.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:34 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/193280079' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-01T02:33:34.727 INFO:teuthology.orchestra.run.smithi062.stderr:adjusted tunables profile to default 2024-06-01T02:33:35.274 INFO:tasks.cephadm:Adding mon.a on smithi062 2024-06-01T02:33:35.274 INFO:tasks.cephadm:Adding mon.c on smithi062 2024-06-01T02:33:35.275 INFO:tasks.cephadm:Adding mon.b on smithi146 2024-06-01T02:33:35.275 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi062:172.21.15.62=a;smithi062:[v2:172.21.15.62:3301,v1:172.21.15.62:6790]=c;smithi146:172.21.15.146=b' 2024-06-01T02:33:36.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:35 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/193280079' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-01T02:33:36.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:35 smithi062 ceph-mon[25095]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:37.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:36 smithi062 ceph-mon[25095]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:37.250 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mon update... 2024-06-01T02:33:37.922 DEBUG:teuthology.orchestra.run.smithi062:mon.c> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.c.service 2024-06-01T02:33:37.926 DEBUG:teuthology.orchestra.run.smithi146:mon.b> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.b.service 2024-06-01T02:33:37.928 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-01T02:33:37.928 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T02:33:38.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi062:172.21.15.62=a;smithi062:[v2:172.21.15.62:3301,v1:172.21.15.62:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:33:38.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: Saving service mon spec with placement smithi062:172.21.15.62=a;smithi062:[v2:172.21.15.62:3301,v1:172.21.15.62:6790]=c;smithi146:172.21.15.146=b;count:3 2024-06-01T02:33:38.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:38.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:38.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:38.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:38.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: Deploying daemon mon.c on smithi062 2024-06-01T02:33:38.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:38 smithi062 ceph-mon[25095]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:39.847 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:33:39.847 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":1,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","modified":"2024-06-01T02:32:11.160426Z","created":"2024-06-01T02:32:11.160426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-01T02:33:39.849 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 1 2024-06-01T02:33:41.373 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-01T02:33:41.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T02:33:46.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:46.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: mon.a calling monitor election 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: mon.c calling monitor election 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: monmap e2: 2 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: fsmap 2024-06-01T02:33:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:46.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: mgrmap e13: y(active, since 36s) 2024-06-01T02:33:46.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: overall HEALTH_OK 2024-06-01T02:33:46.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:46.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:46.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:46.214 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:33:46.214 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","modified":"2024-06-01T02:33:40.553637Z","created":"2024-06-01T02:32:11.160426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-01T02:33:46.216 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 2 2024-06-01T02:33:47.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[25095]: Deploying daemon mon.b on smithi146 2024-06-01T02:33:47.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[25095]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:47.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/2565168462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T02:33:47.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:47.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[29843]: Deploying daemon mon.b on smithi146 2024-06-01T02:33:47.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[29843]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:47.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/2565168462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T02:33:47.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:47.785 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-01T02:33:47.785 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T02:33:49.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:48 smithi062 ceph-mon[25095]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:49.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:48 smithi062 ceph-mon[29843]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:49.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:49 smithi146 systemd[1]: Starting Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:33:50.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 podman[25892]: 2024-06-01 02:33:50.148424069 +0000 UTC m=+0.197598402 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-06-01T02:33:50.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 podman[25892]: 2024-06-01 02:33:50.250380046 +0000 UTC m=+0.299554412 container create 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, RELEASE=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vcs-type=git, ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, name=centos-stream, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream) 2024-06-01T02:33:50.392 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:33:50.392 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":2,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","modified":"2024-06-01T02:33:40.553637Z","created":"2024-06-01T02:32:11.160426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-01T02:33:50.393 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 2 2024-06-01T02:33:50.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 podman[25892]: 2024-06-01 02:33:50.486758982 +0000 UTC m=+0.535933303 container init 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=) 2024-06-01T02:33:50.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 podman[25892]: 2024-06-01 02:33:50.490621887 +0000 UTC m=+0.539796200 container start 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , RELEASE=HEAD, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-06-01T02:33:50.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: pidfile_write: ignore empty --pid-file 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: load: jerasure load: lrc 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: RocksDB version: 6.15.5 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Compile date Apr 18 2022 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: DB SUMMARY 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: DB Session ID: DNSAHH8TVKB3P6JE3GVD 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: CURRENT file: CURRENT 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-06-01T02:33:50.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.error_if_exists: 0 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.create_if_missing: 0 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.paranoid_checks: 1 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.env: 0x5558d6ac6860 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.fs: Posix File System 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.info_log: 0x5558d9089dc0 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T02:33:50.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.statistics: (nil) 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.use_fsync: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_log_file_size: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.allow_fallocate: 1 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.use_direct_reads: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T02:33:50.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.db_log_dir: 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T02:33:50.708 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T02:33:50.709 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.write_buffer_manager: 0x5558d917a240 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.unordered_write: 0 2024-06-01T02:33:50.710 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.row_cache: None 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.wal_filter: None 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.preserve_deletes: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.two_write_queues: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.atomic_flush: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T02:33:50.711 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.log_readahead_size: 0 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_background_jobs: 2 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_background_compactions: -1 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_subcompactions: 1 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T02:33:50.712 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_open_files: -1 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_background_flushes: -1 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Compression algorithms supported: 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T02:33:50.713 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kZSTD supported: 0 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kXpressCompression supported: 0 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kLZ4Compression supported: 1 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kBZip2Compression supported: 0 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kZlibCompression supported: 1 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: kSnappyCompression supported: 1 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.merge_operator: 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_filter: None 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T02:33:50.714 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5558d9055d00) 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_type: 0 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_index_type: 0 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_shortening: 1 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: hash_index_allow_collision: 1 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: checksum: 1 2024-06-01T02:33:50.715 INFO:journalctl@ceph.mon.b.smithi146.stdout: no_block_cache: 0 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache: 0x5558d90c0170 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_options: 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: capacity : 536870912 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_shard_bits : 4 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: strict_capacity_limit : 0 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_compressed: (nil) 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: persistent_cache: (nil) 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size: 4096 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size_deviation: 10 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_restart_interval: 16 2024-06-01T02:33:50.716 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_block_restart_interval: 1 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: metadata_block_size: 4096 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: partition_filters: 0 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_delta_encoding: 1 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: whole_key_filtering: 1 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: verify_compression: 0 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: format_version: 4 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_index_compression: 1 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_align: 0 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression: NoCompression 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T02:33:50.717 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.num_levels: 7 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T02:33:50.718 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T02:33:50.719 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T02:33:50.720 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.arena_block_size: 4194304 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T02:33:50.721 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.table_properties_collectors: 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.inplace_update_support: 0 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T02:33:50.722 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.bloom_locality: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.max_successive_merges: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.ttl: 2592000 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.enable_blob_files: false 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.min_blob_size: 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-01T02:33:50.723 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209230517530, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209230517830, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1717209230, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "39dce86c-dbd9-43b6-9941-49aae21c4be3", "db_session_id": "DNSAHH8TVKB3P6JE3GVD"}} 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209230518139, "job": 1, "event": "recovery_finished"} 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5558d90a6a80 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: DB pointer 0x5558d90b6000 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** DB Stats ** 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:33:50.724 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.725 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:33:50.726 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-01T02:33:50.727 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: using public_addr v2:172.21.15.146:0/0 -> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: starting mon.b rank -1 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:33:50.728 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(???) e0 preinit fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).mds e1 new map 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).mds e1 print_map 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout: e1 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout: legacy client fscid: -1 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout: No filesystems configured 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T02:33:50.729 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi062:172.21.15.62=a;smithi062:[v2:172.21.15.62:3301,v1:172.21.15.62:6790]=c;smithi146:172.21.15.146=b", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: Saving service mon spec with placement smithi062:172.21.15.62=a;smithi062:[v2:172.21.15.62:3301,v1:172.21.15.62:6790]=c;smithi146:172.21.15.146=b;count:3 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:50.730 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: Deploying daemon mon.c on smithi062 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.a calling monitor election 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.c calling monitor election 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.731 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: monmap e2: 2 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: fsmap 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mgrmap e13: y(active, since 36s) 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: overall HEALTH_OK 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: Deploying daemon mon.b on smithi146 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/2565168462' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:50.732 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-06-01T02:33:50.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 ceph-mon[25945]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-06-01T02:33:50.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 bash[25892]: 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a 2024-06-01T02:33:50.733 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:50 smithi146 systemd[1]: Started Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:33:52.194 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-06-01T02:33:52.194 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: mon.a calling monitor election 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: mon.c calling monitor election 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: monmap e3: 3 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: fsmap 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: mgrmap e13: y(active, since 46s) 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: mon.a calling monitor election 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: mon.c calling monitor election 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: monmap e3: 3 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: fsmap 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: mgrmap e13: y(active, since 46s) 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:56.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:56.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[25095]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[29843]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:56.727 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:56 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.019 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-06-01T02:33:57.019 INFO:teuthology.orchestra.run.smithi146.stdout:{"epoch":3,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","modified":"2024-06-01T02:33:50.539528Z","created":"2024-06-01T02:32:11.160426Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3300","nonce":0},{"type":"v1","addr":"172.21.15.62:6789","nonce":0}]},"addr":"172.21.15.62:6789/0","public_addr":"172.21.15.62:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:3301","nonce":0},{"type":"v1","addr":"172.21.15.62:6790","nonce":0}]},"addr":"172.21.15.62:6790/0","public_addr":"172.21.15.62:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:3300","nonce":0},{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-01T02:33:57.021 INFO:teuthology.orchestra.run.smithi146.stderr:dumped monmap epoch 3 2024-06-01T02:33:57.678 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.a calling monitor election 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.c calling monitor election 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-06-01T02:33:57.679 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: monmap e3: 3 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: fsmap 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mgrmap e13: y(active, since 46s) 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.b (rank 2) addr [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] is down (out of quorum) 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:57.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.b calling monitor election 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.b calling monitor election 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.c calling monitor election 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.a calling monitor election 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: monmap e3: 3 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: fsmap 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: mgrmap e13: y(active, since 48s) 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: Cluster is now healthy 2024-06-01T02:33:57.681 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:57 smithi146 ceph-mon[25945]: overall HEALTH_OK 2024-06-01T02:33:57.741 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-01T02:33:57.741 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: mon.b calling monitor election 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: mon.b calling monitor election 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: mon.c calling monitor election 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: mon.a calling monitor election 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: monmap e3: 3 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: fsmap 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: mgrmap e13: y(active, since 48s) 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: Cluster is now healthy 2024-06-01T02:33:57.936 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[25095]: overall HEALTH_OK 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: mon.b calling monitor election 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: mon.b calling monitor election 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: mon.c calling monitor election 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: mon.a calling monitor election 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: monmap e3: 3 mons at {a=[v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0],b=[v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0],c=[v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0]} 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: fsmap 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: osdmap e4: 0 total, 0 up, 0 in 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: mgrmap e13: y(active, since 48s) 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: Cluster is now healthy 2024-06-01T02:33:57.937 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:57 smithi062 ceph-mon[29843]: overall HEALTH_OK 2024-06-01T02:33:59.179 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: Reconfiguring mon.c (monmap changed)... 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: Reconfiguring daemon mon.c on smithi062 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:59.180 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: Reconfiguring mon.c (monmap changed)... 2024-06-01T02:33:59.181 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:59.181 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:33:59.181 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:59.181 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: Reconfiguring daemon mon.c on smithi062 2024-06-01T02:33:59.181 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:33:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:33:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: Reconfiguring mon.c (monmap changed)... 2024-06-01T02:33:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:33:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:33:59.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:33:59.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: Reconfiguring daemon mon.c on smithi062 2024-06-01T02:33:59.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:33:59.578 INFO:teuthology.orchestra.run.smithi062.stdout:# minimal ceph.conf for 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:33:59.578 INFO:teuthology.orchestra.run.smithi062.stdout:[global] 2024-06-01T02:33:59.578 INFO:teuthology.orchestra.run.smithi062.stdout: fsid = 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:33:59.578 INFO:teuthology.orchestra.run.smithi062.stdout: mon_host = [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] 2024-06-01T02:34:00.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:59 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4170213269' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:00.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:33:59 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:00.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:59 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4170213269' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:00.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:33:59 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:00.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:59 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4170213269' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:00.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:33:59 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:00.920 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-01T02:34:00.920 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:34:00.920 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.conf 2024-06-01T02:34:00.949 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:34:00.949 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:34:01.022 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:34:01.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.conf 2024-06-01T02:34:01.055 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:34:01.056 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:34:01.122 INFO:tasks.cephadm:Adding mgr.y on smithi062 2024-06-01T02:34:01.122 INFO:tasks.cephadm:Adding mgr.x on smithi146 2024-06-01T02:34:01.122 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '2;smithi062=y;smithi146=x' 2024-06-01T02:34:01.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:00 smithi146 ceph-mon[25945]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:01.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:01.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:34:01.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:34:01.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:01.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[25095]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:01.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:01.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:34:01.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:34:01.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[29843]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:34:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:34:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:02 smithi146 ceph-mon[25945]: Reconfiguring mon.b (monmap changed)... 2024-06-01T02:34:02.261 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:02 smithi146 ceph-mon[25945]: Reconfiguring daemon mon.b on smithi146 2024-06-01T02:34:02.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:02 smithi062 ceph-mon[25095]: Reconfiguring mon.b (monmap changed)... 2024-06-01T02:34:02.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:02 smithi062 ceph-mon[25095]: Reconfiguring daemon mon.b on smithi146 2024-06-01T02:34:02.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:02 smithi062 ceph-mon[29843]: Reconfiguring mon.b (monmap changed)... 2024-06-01T02:34:02.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:02 smithi062 ceph-mon[29843]: Reconfiguring daemon mon.b on smithi146 2024-06-01T02:34:03.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[25095]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:03.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:03.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:03.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[29843]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:03 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:03.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:03 smithi146 ceph-mon[25945]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:03.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:03 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:03.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:03 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:03.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:03 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:03.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:03 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:03.326 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:03 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:03.540 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled mgr update... 2024-06-01T02:34:04.175 DEBUG:teuthology.orchestra.run.smithi146:mgr.x> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service 2024-06-01T02:34:04.218 INFO:tasks.cephadm:Deploying OSDs... 2024-06-01T02:34:04.218 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:34:04.218 DEBUG:teuthology.orchestra.run.smithi062:> dd if=/scratch_devs of=/dev/stdout 2024-06-01T02:34:04.239 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-01T02:34:04.239 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_1 2024-06-01T02:34:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-01T02:34:04.299 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:04.299 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-01T02:34:04.300 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:04.300 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:04.300 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-06-01 02:33:40.010537619 +0000 2024-06-01T02:34:04.300 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-06-01 02:31:06.139138262 +0000 2024-06-01T02:34:04.300 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-06-01 02:31:06.139138262 +0000 2024-06-01T02:34:04.300 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-06-01 02:31:06.139138262 +0000 2024-06-01T02:34:04.300 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-01T02:34:04.368 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-06-01T02:34:04.368 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-06-01T02:34:04.369 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000288836 s, 1.8 MB/s 2024-06-01T02:34:04.370 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-01T02:34:04.429 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_2 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-06-01 02:33:40.011537602 +0000 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-06-01 02:31:06.500132334 +0000 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-06-01 02:31:06.500132334 +0000 2024-06-01T02:34:04.488 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-06-01 02:31:06.500132334 +0000 2024-06-01T02:34:04.488 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-01T02:34:04.555 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-06-01T02:34:04.555 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-06-01T02:34:04.555 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.00021177 s, 2.4 MB/s 2024-06-01T02:34:04.557 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-01T02:34:04.616 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_3 2024-06-01T02:34:04.673 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-06-01 02:33:40.011537602 +0000 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-06-01 02:31:06.850126586 +0000 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-06-01 02:31:06.850126586 +0000 2024-06-01T02:34:04.674 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-06-01 02:31:06.850126586 +0000 2024-06-01T02:34:04.674 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi062=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: Saving service mgr spec with placement smithi062=y;smithi146=x;count:2 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:34:04.689 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:04.690 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: Deploying daemon mgr.x on smithi146 2024-06-01T02:34:04.690 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:04 smithi146 ceph-mon[25945]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:04.738 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi062=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:04.738 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: Saving service mgr spec with placement smithi062=y;smithi146=x;count:2 2024-06-01T02:34:04.738 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:04.738 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:04.738 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: Deploying daemon mgr.x on smithi146 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[29843]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='client.24104 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi062=y;smithi146=x", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: Saving service mgr spec with placement smithi062=y;smithi146=x;count:2 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:04.739 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: Deploying daemon mgr.x on smithi146 2024-06-01T02:34:04.740 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:04 smithi062 ceph-mon[25095]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:04.741 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-06-01T02:34:04.741 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-06-01T02:34:04.741 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000181065 s, 2.8 MB/s 2024-06-01T02:34:04.742 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-01T02:34:04.800 DEBUG:teuthology.orchestra.run.smithi062:> stat /dev/vg_nvme/lv_4 2024-06-01T02:34:04.857 INFO:teuthology.orchestra.run.smithi062.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-01T02:34:04.857 INFO:teuthology.orchestra.run.smithi062.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout:Access: 2024-06-01 02:33:40.012537586 +0000 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout:Modify: 2024-06-01 02:31:07.195120921 +0000 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout:Change: 2024-06-01 02:31:07.195120921 +0000 2024-06-01T02:34:04.858 INFO:teuthology.orchestra.run.smithi062.stdout: Birth: 2024-06-01 02:31:07.195120921 +0000 2024-06-01T02:34:04.859 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-01T02:34:04.924 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records in 2024-06-01T02:34:04.924 INFO:teuthology.orchestra.run.smithi062.stderr:1+0 records out 2024-06-01T02:34:04.924 INFO:teuthology.orchestra.run.smithi062.stderr:512 bytes copied, 0.000195998 s, 2.6 MB/s 2024-06-01T02:34:04.925 DEBUG:teuthology.orchestra.run.smithi062:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-01T02:34:04.984 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:34:04.984 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-06-01T02:34:05.001 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-01T02:34:05.001 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-06-01T02:34:05.057 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-01T02:34:05.058 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:05.058 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-06-01T02:34:05.058 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:05.058 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:05.058 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-06-01 02:33:49.256352735 +0000 2024-06-01T02:34:05.059 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-06-01 02:30:56.121403044 +0000 2024-06-01T02:34:05.059 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-06-01 02:30:56.121403044 +0000 2024-06-01T02:34:05.059 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-06-01 02:30:56.121403044 +0000 2024-06-01T02:34:05.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-01T02:34:05.123 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-06-01T02:34:05.123 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-06-01T02:34:05.123 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000220042 s, 2.3 MB/s 2024-06-01T02:34:05.125 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-01T02:34:05.181 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-06-01 02:33:49.256352735 +0000 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-06-01 02:30:56.462397877 +0000 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-06-01 02:30:56.462397877 +0000 2024-06-01T02:34:05.238 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-06-01 02:30:56.462397877 +0000 2024-06-01T02:34:05.239 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-01T02:34:05.301 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-06-01T02:34:05.302 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-06-01T02:34:05.302 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000172205 s, 3.0 MB/s 2024-06-01T02:34:05.303 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-01T02:34:05.359 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-06-01T02:34:05.415 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-06-01 02:33:49.256352735 +0000 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-06-01 02:30:56.797392801 +0000 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-06-01 02:30:56.797392801 +0000 2024-06-01T02:34:05.416 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-06-01 02:30:56.797392801 +0000 2024-06-01T02:34:05.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-01T02:34:05.494 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-06-01T02:34:05.494 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-06-01T02:34:05.494 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000176482 s, 2.9 MB/s 2024-06-01T02:34:05.496 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-01T02:34:05.553 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-06-01T02:34:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-01T02:34:05.609 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-01T02:34:05.609 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-06-01T02:34:05.609 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-01T02:34:05.609 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-06-01T02:34:05.610 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-06-01 02:34:05.454108030 +0000 2024-06-01T02:34:05.610 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-06-01 02:30:57.112388028 +0000 2024-06-01T02:34:05.610 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-06-01 02:30:57.112388028 +0000 2024-06-01T02:34:05.610 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-06-01 02:30:57.112388028 +0000 2024-06-01T02:34:05.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-01T02:34:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-06-01T02:34:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-06-01T02:34:05.675 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000229026 s, 2.2 MB/s 2024-06-01T02:34:05.676 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-01T02:34:05.735 INFO:tasks.cephadm:Deploying osd.0 on smithi062 with /dev/vg_nvme/lv_4... 2024-06-01T02:34:05.735 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-01T02:34:05.954 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:05 smithi146 podman[27105]: 2024-06-01 02:34:05.909111597 +0000 UTC m=+0.119303364 container create 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, ceph=True, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-06-01T02:34:06.269 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 podman[27105]: 2024-06-01 02:34:06.095429081 +0000 UTC m=+0.305620827 container init 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, name=centos-stream, architecture=x86_64, version=8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-type=git, io.openshift.expose-services=) 2024-06-01T02:34:06.269 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 podman[27105]: 2024-06-01 02:34:06.098295691 +0000 UTC m=+0.308487441 container start 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git) 2024-06-01T02:34:06.269 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-mgr[27124]: -- 172.21.15.146:0/3120777639 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ab5e544340 con 0x55ab5e51c800 2024-06-01T02:34:06.270 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 bash[27105]: 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d 2024-06-01T02:34:06.270 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 systemd[1]: Started Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:34:06.270 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:06.249+0000 7f47f9d2e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:34:06.701 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:06.385+0000 7f47f9d2e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:34:06.701 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:06.494+0000 7f47f9d2e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:34:06.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-mon[25945]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:06.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:06.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:06.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:06.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:06.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:06.826+0000 7f47f9d2e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:34:06.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:06.910+0000 7f47f9d2e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:34:07.067 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[25095]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:07.067 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:07.067 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:07.067 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:07.067 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:07.068 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[29843]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:07.068 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:07.068 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:07.068 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:07.068 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:06 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:07.839 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-01T02:34:07.840 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-01T02:34:07.840 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-06-01T02:34:07.840 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-06-01T02:34:07.840 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0115503 s, 908 MB/s 2024-06-01T02:34:07.840 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 2024-06-01T02:34:07.840 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-06-01T02:34:08.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: Reconfiguring mgr.y (unknown last config time)... 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: Reconfiguring daemon mgr.y on smithi062 2024-06-01T02:34:08.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[25095]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: Reconfiguring mgr.y (unknown last config time)... 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: Reconfiguring daemon mgr.y on smithi062 2024-06-01T02:34:08.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:08 smithi062 ceph-mon[29843]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:08.454 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:08.049+0000 7f47f9d2e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:34:08.454 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:08.211+0000 7f47f9d2e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:34:08.454 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:08.291+0000 7f47f9d2e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: Reconfiguring mgr.y (unknown last config time)... 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: Reconfiguring daemon mgr.y on smithi062 2024-06-01T02:34:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-mon[25945]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:08.703 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_4 2024-06-01T02:34:08.954 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:08.599+0000 7f47f9d2e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:34:08.954 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:08.687+0000 7f47f9d2e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:34:08.954 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:08.778+0000 7f47f9d2e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:34:09.448 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:09.015+0000 7f47f9d2e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:34:09.704 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:09.449+0000 7f47f9d2e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:34:10.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:09.927+0000 7f47f9d2e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:34:10.218 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:10.018+0000 7f47f9d2e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:34:10.219 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.237 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:34:10.238 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:10.704 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:10.307+0000 7f47f9d2e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:34:10.704 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:10.394+0000 7f47f9d2e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:34:11.124 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:10.794+0000 7f47f9d2e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:34:11.124 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:10.926+0000 7f47f9d2e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:34:11.124 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:11.017+0000 7f47f9d2e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:34:11.339 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[25095]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:11.339 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:11.339 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:11.339 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:11.340 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[29843]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:11.340 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:11.340 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:11.340 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:11 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:11.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-mon[25945]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:11.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:11.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:11.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:11.454 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:11.133+0000 7f47f9d2e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:34:11.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:11.222+0000 7f47f9d2e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:34:12.129 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:11.768+0000 7f47f9d2e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:34:12.333 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:12 smithi062 ceph-mon[25095]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:12.334 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:12 smithi062 ceph-mon[29843]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:12.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:12 smithi146 ceph-mon[25945]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:12.454 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:34:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:34:12.131+0000 7f47f9d2e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:34:13.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:13 smithi146 ceph-mon[25945]: Standby manager daemon x started 2024-06-01T02:34:13.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:13 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:34:13.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:13 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:34:13.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:13 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:34:13.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:13 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:34:13.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[25095]: Standby manager daemon x started 2024-06-01T02:34:13.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:34:13.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[29843]: Standby manager daemon x started 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:34:13.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:13 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/175170811' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:34:14.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: mgrmap e14: y(active, since 63s), standbys: x 2024-06-01T02:34:14.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:34:14.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2891090681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e376b0d-882e-48c6-b47f-8d68368ff47b"}]: dispatch 2024-06-01T02:34:14.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2891090681' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e376b0d-882e-48c6-b47f-8d68368ff47b"}]': finished 2024-06-01T02:34:14.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: osdmap e5: 1 total, 0 up, 1 in 2024-06-01T02:34:14.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:14.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:14.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:14 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2523195974' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:34:14.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: mgrmap e14: y(active, since 63s), standbys: x 2024-06-01T02:34:14.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:34:14.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2891090681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e376b0d-882e-48c6-b47f-8d68368ff47b"}]: dispatch 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2891090681' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e376b0d-882e-48c6-b47f-8d68368ff47b"}]': finished 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: osdmap e5: 1 total, 0 up, 1 in 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2523195974' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: mgrmap e14: y(active, since 63s), standbys: x 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2891090681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e376b0d-882e-48c6-b47f-8d68368ff47b"}]: dispatch 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2891090681' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e376b0d-882e-48c6-b47f-8d68368ff47b"}]': finished 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: osdmap e5: 1 total, 0 up, 1 in 2024-06-01T02:34:14.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:14.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:14.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:14 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2523195974' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:34:16.942 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:16 smithi062 ceph-mon[25095]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:16.943 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:16 smithi062 ceph-mon[29843]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:17.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:16 smithi146 ceph-mon[25945]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:18.998 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:18 smithi062 ceph-mon[25095]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:18.998 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:18 smithi062 ceph-mon[29843]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:19.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:18 smithi146 ceph-mon[25945]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:20.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:34:20.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:20.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:34:20.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:20.154 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:34:20.155 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:20.953 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:20 smithi062 ceph-mon[29843]: Deploying daemon osd.0 on smithi062 2024-06-01T02:34:20.953 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:20 smithi062 ceph-mon[29843]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:20.953 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:20 smithi062 ceph-mon[25095]: Deploying daemon osd.0 on smithi062 2024-06-01T02:34:20.953 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:20 smithi062 ceph-mon[25095]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:21.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:20 smithi146 ceph-mon[25945]: Deploying daemon osd.0 on smithi062 2024-06-01T02:34:21.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:20 smithi146 ceph-mon[25945]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:23.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:22 smithi146 ceph-mon[25945]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:23.234 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:22 smithi062 ceph-mon[25095]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:23.234 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:22 smithi062 ceph-mon[29843]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:25.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[25095]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:25.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:25.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[29843]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:24 smithi146 ceph-mon[25945]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:27.043 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:26 smithi062 ceph-mon[25095]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:27.043 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:26 smithi062 ceph-mon[29843]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:27.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:26 smithi146 ceph-mon[25945]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:27.888 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 0 on host 'smithi062' 2024-06-01T02:34:28.210 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.210 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:34:28.210 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.210 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.210 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:28.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:28.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:34:28.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:28.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:28.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:28.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:28.502 DEBUG:teuthology.orchestra.run.smithi062:osd.0> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0.service 2024-06-01T02:34:28.504 INFO:tasks.cephadm:Deploying osd.1 on smithi062 with /dev/vg_nvme/lv_3... 2024-06-01T02:34:28.505 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-01T02:34:29.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[25095]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:29.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[25095]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T02:34:29.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[25095]: osdmap e6: 1 total, 0 up, 1 in 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[25095]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[29843]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[29843]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[29843]: osdmap e6: 1 total, 0 up, 1 in 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:29.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-mon[29843]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:29.262 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[32673]: 2024-06-01T02:34:29.032+0000 7f5301f71700 -1 osd.0 0 waiting for initial osdmap 2024-06-01T02:34:29.263 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:34:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[32673]: 2024-06-01T02:34:29.038+0000 7f52fa904700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:34:29.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:29 smithi146 ceph-mon[25945]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:29.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:29 smithi146 ceph-mon[25945]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T02:34:29.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:29 smithi146 ceph-mon[25945]: osdmap e6: 1 total, 0 up, 1 in 2024-06-01T02:34:29.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:29 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:29.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:29 smithi146 ceph-mon[25945]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:30.330 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[25095]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:34:30.331 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[25095]: osdmap e7: 1 total, 0 up, 1 in 2024-06-01T02:34:30.331 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:30.331 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:30.331 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[29843]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:34:30.331 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[29843]: osdmap e7: 1 total, 0 up, 1 in 2024-06-01T02:34:30.332 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:30.332 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:30 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:30.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:30 smithi146 ceph-mon[25945]: from='osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:34:30.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:30 smithi146 ceph-mon[25945]: osdmap e7: 1 total, 0 up, 1 in 2024-06-01T02:34:30.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:30 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:30.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:30 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:31.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760] boot 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: osdmap e8: 1 total, 1 up, 1 in 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:31.316 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:31.317 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760] boot 2024-06-01T02:34:31.317 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: osdmap e8: 1 total, 1 up, 1 in 2024-06-01T02:34:31.317 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:31 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:31.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:34:31.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:34:31.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-01T02:34:31.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:31.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: osd.0 [v2:172.21.15.62:6802/2961277760,v1:172.21.15.62:6803/2961277760] boot 2024-06-01T02:34:31.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: osdmap e8: 1 total, 1 up, 1 in 2024-06-01T02:34:31.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:31 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:34:32.310 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:32 smithi062 ceph-mon[25095]: osdmap e9: 1 total, 1 up, 1 in 2024-06-01T02:34:32.310 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:32 smithi062 ceph-mon[25095]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:32.310 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:32 smithi062 ceph-mon[29843]: osdmap e9: 1 total, 1 up, 1 in 2024-06-01T02:34:32.310 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:32 smithi062 ceph-mon[29843]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:32.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:32 smithi146 ceph-mon[25945]: osdmap e9: 1 total, 1 up, 1 in 2024-06-01T02:34:32.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:32 smithi146 ceph-mon[25945]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-01T02:34:32.626 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-01T02:34:32.627 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-06-01T02:34:32.627 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-06-01T02:34:32.627 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113139 s, 927 MB/s 2024-06-01T02:34:32.627 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-06-01T02:34:33.239 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi062 2024-06-01T02:34:33.239 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi062 to 16064M 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi062 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi062 to 16064M 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.240 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.269 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_3 2024-06-01T02:34:33.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:33 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi062 2024-06-01T02:34:33.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:33.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:33 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi062 to 16064M 2024-06-01T02:34:33.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:33.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:35.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:34 smithi062 ceph-mon[25095]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:35.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:34 smithi062 ceph-mon[29843]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:35.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:34 smithi146 ceph-mon[25945]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:35.851 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:35 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:35.851 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:35 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:35.852 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:35 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:35.852 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:35 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:35.852 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:35 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:35.852 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:35 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:36.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:35 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:36.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:35 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:36.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:35 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:37.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:36 smithi062 ceph-mon[29843]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:37.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:36 smithi062 ceph-mon[29843]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:37.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:36 smithi062 ceph-mon[25095]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:37.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:36 smithi062 ceph-mon[25095]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:37.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:36 smithi146 ceph-mon[25945]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:37.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:36 smithi146 ceph-mon[25945]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:38.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:37 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3532851387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4"}]: dispatch 2024-06-01T02:34:38.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:37 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3532851387' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4"}]': finished 2024-06-01T02:34:38.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:37 smithi146 ceph-mon[25945]: osdmap e10: 2 total, 1 up, 2 in 2024-06-01T02:34:38.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:37 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:38.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3532851387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4"}]: dispatch 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3532851387' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4"}]': finished 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[29843]: osdmap e10: 2 total, 1 up, 2 in 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3532851387' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4"}]: dispatch 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3532851387' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4"}]': finished 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[25095]: osdmap e10: 2 total, 1 up, 2 in 2024-06-01T02:34:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:37 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:38 smithi146 ceph-mon[25945]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:38 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/89301224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:34:39.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:38 smithi062 ceph-mon[25095]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:39.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:38 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/89301224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:34:39.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:38 smithi062 ceph-mon[29843]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:39.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:38 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/89301224' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:34:40.972 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:40 smithi062 ceph-mon[25095]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:40.973 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:40 smithi062 ceph-mon[29843]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:41.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:40 smithi146 ceph-mon[25945]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:43.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:42 smithi062 ceph-mon[25095]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:43.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:42 smithi062 ceph-mon[29843]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:43.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:42 smithi146 ceph-mon[25945]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:44.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:43 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:34:44.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:43 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:44.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:43 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:34:44.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:43 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:44.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:43 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:34:44.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:43 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:45.139 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:44 smithi062 ceph-mon[25095]: Deploying daemon osd.1 on smithi062 2024-06-01T02:34:45.139 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:44 smithi062 ceph-mon[25095]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:45.139 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:44 smithi062 ceph-mon[29843]: Deploying daemon osd.1 on smithi062 2024-06-01T02:34:45.140 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:44 smithi062 ceph-mon[29843]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:45.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:44 smithi146 ceph-mon[25945]: Deploying daemon osd.1 on smithi062 2024-06-01T02:34:45.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:44 smithi146 ceph-mon[25945]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:47.080 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:46 smithi062 ceph-mon[25095]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:47.080 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:46 smithi062 ceph-mon[29843]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:47.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:46 smithi146 ceph-mon[25945]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:49.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:48 smithi062 ceph-mon[25095]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:49.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:48 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:49.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:48 smithi062 ceph-mon[29843]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:49.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:48 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:49.115 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:48 smithi146 ceph-mon[25945]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:49.115 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:48 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:50.125 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:49 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:50.125 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:49 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:50.125 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:49 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:50.125 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:49 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:50.125 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:49 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:50.125 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:49 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:50.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:49 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:50.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:49 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:50.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:49 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:51.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:50 smithi146 ceph-mon[25945]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:51.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:50 smithi062 ceph-mon[25095]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:51.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:50 smithi062 ceph-mon[29843]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:51.860 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 1 on host 'smithi062' 2024-06-01T02:34:52.468 DEBUG:teuthology.orchestra.run.smithi062:osd.1> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.1.service 2024-06-01T02:34:52.471 INFO:tasks.cephadm:Deploying osd.2 on smithi062 with /dev/vg_nvme/lv_2... 2024-06-01T02:34:52.471 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-01T02:34:52.968 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[25095]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[25095]: from='osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[25095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[29843]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[29843]: from='osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[29843]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:34:52.969 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:52 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:53.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:52 smithi146 ceph-mon[25945]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:53.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:52 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:53.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:52 smithi146 ceph-mon[25945]: from='osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:34:53.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:52 smithi146 ceph-mon[25945]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:34:53.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:52 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:54.165 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[29843]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[29843]: osdmap e11: 2 total, 1 up, 2 in 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[29843]: from='osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[29843]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[25095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[25095]: osdmap e11: 2 total, 1 up, 2 in 2024-06-01T02:34:54.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:54.167 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[25095]: from='osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:54.167 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-mon[25095]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:54.167 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[35614]: 2024-06-01T02:34:53.892+0000 7f3114c1a700 -1 osd.1 0 waiting for initial osdmap 2024-06-01T02:34:54.167 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:34:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[35614]: 2024-06-01T02:34:53.894+0000 7f310edb0700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:34:54.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:53 smithi146 ceph-mon[25945]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T02:34:54.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:53 smithi146 ceph-mon[25945]: osdmap e11: 2 total, 1 up, 2 in 2024-06-01T02:34:54.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:53 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:54.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:53 smithi146 ceph-mon[25945]: from='osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:54.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:53 smithi146 ceph-mon[25945]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:34:55.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:54 smithi146 ceph-mon[25945]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:55.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:54 smithi146 ceph-mon[25945]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:34:55.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:54 smithi146 ceph-mon[25945]: osdmap e12: 2 total, 1 up, 2 in 2024-06-01T02:34:55.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:55.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:55.210 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[25095]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[25095]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[25095]: osdmap e12: 2 total, 1 up, 2 in 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[29843]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[29843]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[29843]: osdmap e12: 2 total, 1 up, 2 in 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:55.211 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:55.715 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-01T02:34:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-01T02:34:55.716 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-06-01T02:34:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-06-01T02:34:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0151241 s, 693 MB/s 2024-06-01T02:34:55.716 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-06-01T02:34:56.158 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:34:56.158 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:34:56.158 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:56.158 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[25095]: osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874] boot 2024-06-01T02:34:56.158 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[25095]: osdmap e13: 2 total, 2 up, 2 in 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[29843]: osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874] boot 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[29843]: osdmap e13: 2 total, 2 up, 2 in 2024-06-01T02:34:56.159 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:56.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:55 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:34:56.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:55 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:34:56.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:55 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:56.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:55 smithi146 ceph-mon[25945]: osd.1 [v2:172.21.15.62:6810/42392874,v1:172.21.15.62:6811/42392874] boot 2024-06-01T02:34:56.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:55 smithi146 ceph-mon[25945]: osdmap e13: 2 total, 2 up, 2 in 2024-06-01T02:34:56.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:55 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:34:56.354 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_2 2024-06-01T02:34:56.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:56 smithi062 ceph-mon[25095]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:56.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:56 smithi062 ceph-mon[25095]: osdmap e14: 2 total, 2 up, 2 in 2024-06-01T02:34:56.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:56 smithi062 ceph-mon[29843]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:56.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:56 smithi062 ceph-mon[29843]: osdmap e14: 2 total, 2 up, 2 in 2024-06-01T02:34:57.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:56 smithi146 ceph-mon[25945]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-06-01T02:34:57.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:56 smithi146 ceph-mon[25945]: osdmap e14: 2 total, 2 up, 2 in 2024-06-01T02:34:58.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi062 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi062 to 8032M 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:58 smithi146 ceph-mon[25945]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:34:58.978 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi062 2024-06-01T02:34:58.978 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.978 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:58.978 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi062 to 8032M 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[25095]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi062 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.979 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi062 to 8032M 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:34:58.980 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:58 smithi062 ceph-mon[29843]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:34:59.942 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[25095]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:59.942 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:59.943 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:59.943 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:59.943 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[29843]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:59.943 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:59.943 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:59.943 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:34:59 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:34:59.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:59 smithi146 ceph-mon[25945]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:34:59.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:59 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:34:59.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:59 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:34:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:34:59 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:00.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:00 smithi146 ceph-mon[25945]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:01.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:00 smithi062 ceph-mon[25095]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:01.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:00 smithi062 ceph-mon[29843]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:01.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:01 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3350056020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c16629b6-e552-407a-b74b-d2beb921a2df"}]: dispatch 2024-06-01T02:35:01.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:01 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3350056020' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c16629b6-e552-407a-b74b-d2beb921a2df"}]': finished 2024-06-01T02:35:01.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:01 smithi146 ceph-mon[25945]: osdmap e15: 3 total, 2 up, 3 in 2024-06-01T02:35:01.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:01 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:01.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:01 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1480239291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3350056020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c16629b6-e552-407a-b74b-d2beb921a2df"}]: dispatch 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3350056020' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c16629b6-e552-407a-b74b-d2beb921a2df"}]': finished 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[25095]: osdmap e15: 3 total, 2 up, 3 in 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1480239291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3350056020' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c16629b6-e552-407a-b74b-d2beb921a2df"}]: dispatch 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3350056020' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c16629b6-e552-407a-b74b-d2beb921a2df"}]': finished 2024-06-01T02:35:02.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[29843]: osdmap e15: 3 total, 2 up, 3 in 2024-06-01T02:35:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:01 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1480239291' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:02.761 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:02 smithi062 ceph-mon[25095]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:02.761 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:02 smithi062 ceph-mon[29843]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:02.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:02 smithi146 ceph-mon[25945]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:05.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:04 smithi062 ceph-mon[25095]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:05.005 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:04 smithi062 ceph-mon[29843]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:05.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:04 smithi146 ceph-mon[25945]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:07.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:06 smithi062 ceph-mon[25095]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:07.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:06 smithi062 ceph-mon[29843]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:07.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:06 smithi146 ceph-mon[25945]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:08.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:07 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:35:08.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:07 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:08.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:07 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:35:08.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:07 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:08.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:07 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:35:08.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:07 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:09.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:08 smithi062 ceph-mon[25095]: Deploying daemon osd.2 on smithi062 2024-06-01T02:35:09.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:08 smithi062 ceph-mon[25095]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:09.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:08 smithi062 ceph-mon[29843]: Deploying daemon osd.2 on smithi062 2024-06-01T02:35:09.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:08 smithi062 ceph-mon[29843]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:09.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:08 smithi146 ceph-mon[25945]: Deploying daemon osd.2 on smithi062 2024-06-01T02:35:09.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:08 smithi146 ceph-mon[25945]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:11.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:10 smithi062 ceph-mon[25095]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:11.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:35:11.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:35:11.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:10 smithi062 ceph-mon[29843]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:11.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:35:11.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:35:11.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:10 smithi146 ceph-mon[25945]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:11.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:35:11.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:35:12.900 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[25095]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[29843]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:12.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:13.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:12 smithi146 ceph-mon[25945]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:13.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:14.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:14 smithi062 ceph-mon[25095]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:14.976 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:14 smithi062 ceph-mon[29843]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:15.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:14 smithi146 ceph-mon[25945]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:15.612 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 2 on host 'smithi062' 2024-06-01T02:35:15.975 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:15.975 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:35:15.976 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:15.977 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:15.977 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:15.977 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:16.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:16.250 DEBUG:teuthology.orchestra.run.smithi062:osd.2> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.2.service 2024-06-01T02:35:16.252 INFO:tasks.cephadm:Deploying osd.3 on smithi062 with /dev/vg_nvme/lv_1... 2024-06-01T02:35:16.252 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-01T02:35:17.107 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[29843]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:17.107 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[29843]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T02:35:17.107 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[29843]: osdmap e16: 3 total, 2 up, 3 in 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[29843]: from='osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[29843]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[25095]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[25095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[25095]: osdmap e16: 3 total, 2 up, 3 in 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[25095]: from='osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:17.108 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-mon[25095]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:17.109 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[38513]: 2024-06-01T02:35:16.835+0000 7f9c0cb17700 -1 osd.2 0 waiting for initial osdmap 2024-06-01T02:35:17.109 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:16 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[38513]: 2024-06-01T02:35:16.838+0000 7f9c03ca7700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:35:17.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:16 smithi146 ceph-mon[25945]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:16 smithi146 ceph-mon[25945]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T02:35:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:16 smithi146 ceph-mon[25945]: osdmap e16: 3 total, 2 up, 3 in 2024-06-01T02:35:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:16 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:16 smithi146 ceph-mon[25945]: from='osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:16 smithi146 ceph-mon[25945]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:17.900 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[25095]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:35:17.900 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[25095]: osdmap e17: 3 total, 2 up, 3 in 2024-06-01T02:35:17.900 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:17.900 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:17.900 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[29843]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:35:17.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[29843]: osdmap e17: 3 total, 2 up, 3 in 2024-06-01T02:35:17.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:17.901 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:17 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:18.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:17 smithi146 ceph-mon[25945]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:35:18.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:17 smithi146 ceph-mon[25945]: osdmap e17: 3 total, 2 up, 3 in 2024-06-01T02:35:18.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:17 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:17 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651] boot 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: osdmap e18: 3 total, 3 up, 3 in 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:35:18.883 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:35:18.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:18.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:18.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651] boot 2024-06-01T02:35:18.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: osdmap e18: 3 total, 3 up, 3 in 2024-06-01T02:35:18.884 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:18 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:19.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:35:19.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:35:19.106 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-06-01T02:35:19.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:19.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: osd.2 [v2:172.21.15.62:6818/3316357651,v1:172.21.15.62:6819/3316357651] boot 2024-06-01T02:35:19.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: osdmap e18: 3 total, 3 up, 3 in 2024-06-01T02:35:19.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:18 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:35:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-01T02:35:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-01T02:35:20.055 INFO:teuthology.orchestra.run.smithi062.stderr: stderr: 10+0 records in 2024-06-01T02:35:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:10+0 records out 2024-06-01T02:35:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.016557 s, 633 MB/s 2024-06-01T02:35:20.055 INFO:teuthology.orchestra.run.smithi062.stderr:--> Zapping successful for: 2024-06-01T02:35:20.443 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:20 smithi062 ceph-mon[25095]: osdmap e19: 3 total, 3 up, 3 in 2024-06-01T02:35:20.443 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:20 smithi062 ceph-mon[29843]: osdmap e19: 3 total, 3 up, 3 in 2024-06-01T02:35:20.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:20 smithi146 ceph-mon[25945]: osdmap e19: 3 total, 3 up, 3 in 2024-06-01T02:35:20.744 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi062:vg_nvme/lv_1 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi062 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi062 to 5354M 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.314 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: osdmap e20: 3 total, 3 up, 3 in 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi062 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi062 to 5354M 2024-06-01T02:35:21.315 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-01T02:35:21.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: osdmap e20: 3 total, 3 up, 3 in 2024-06-01T02:35:21.316 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:21 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-01T02:35:21.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: pgmap v74: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi062 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi062 to 5354M 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: osdmap e20: 3 total, 3 up, 3 in 2024-06-01T02:35:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:21 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-01T02:35:22.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:22 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-01T02:35:22.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:22 smithi146 ceph-mon[25945]: osdmap e21: 3 total, 3 up, 3 in 2024-06-01T02:35:22.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:22 smithi146 ceph-mon[25945]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:22.747 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:22 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-01T02:35:22.747 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:22 smithi062 ceph-mon[25095]: osdmap e21: 3 total, 3 up, 3 in 2024-06-01T02:35:22.747 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:22 smithi062 ceph-mon[25095]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:22.748 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:22 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-01T02:35:22.748 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:22 smithi062 ceph-mon[29843]: osdmap e21: 3 total, 3 up, 3 in 2024-06-01T02:35:22.748 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:22 smithi062 ceph-mon[29843]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:23.011 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:35:22 smithi062 sudo[39886]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T02:35:23.011 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:35:22 smithi062 sudo[39886]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:23.011 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:35:22 smithi062 sudo[39886]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:23.356 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[29843]: osdmap e22: 3 total, 3 up, 3 in 2024-06-01T02:35:23.356 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[29843]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:35:23.356 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:35:23.357 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:35:23.357 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:23.357 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:23.357 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:35:23 smithi062 sudo[39932]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T02:35:23.357 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:35:23 smithi062 sudo[39932]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:23.357 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:35:23 smithi062 sudo[39932]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:23.357 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[25095]: osdmap e22: 3 total, 3 up, 3 in 2024-06-01T02:35:23.358 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[25095]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:35:23.358 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:35:23.358 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:35:23.358 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:23.358 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:23 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:23.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:23 smithi146 ceph-mon[25945]: osdmap e22: 3 total, 3 up, 3 in 2024-06-01T02:35:23.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:23 smithi146 ceph-mon[25945]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi062:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:35:23.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:23 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:35:23.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:23 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:35:23.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:23 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:23.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:23 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:24.011 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:35:23 smithi062 sudo[39954]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T02:35:24.011 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:35:23 smithi062 sudo[39954]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:24.011 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:35:23 smithi062 sudo[39954]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:24.406 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:35:24 smithi062 sudo[39974]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T02:35:24.406 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:35:24 smithi062 sudo[39974]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:24.406 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:35:24 smithi062 sudo[39974]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:24.747 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:24 smithi062 sudo[39985]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T02:35:24.747 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:24 smithi062 sudo[39985]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:24.748 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:24 smithi062 sudo[39985]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:25.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:24 smithi062 ceph-mon[25095]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:25.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:24 smithi062 ceph-mon[25095]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-01T02:35:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:24 smithi062 ceph-mon[29843]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:25.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:24 smithi062 ceph-mon[29843]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-01T02:35:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:24 smithi146 ceph-mon[25945]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:25.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:24 smithi146 ceph-mon[25945]: mgrmap e15: y(active, since 2m), standbys: x 2024-06-01T02:35:25.511 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40021]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T02:35:25.511 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40021]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:25.511 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40021]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:25.970 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40025]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T02:35:25.970 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40025]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:25.970 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40025]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:26.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40028]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T02:35:26.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40028]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:26.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:25 smithi062 sudo[40028]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:26.761 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 sudo[40031]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T02:35:26.761 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 sudo[40031]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:26.761 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 sudo[40031]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:27.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:27.135 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3882696569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]': finished 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: osdmap e23: 4 total, 3 up, 4 in 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 sudo[27456]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 sudo[27456]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:27.136 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 sudo[27456]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:27.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:27.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3882696569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]: dispatch 2024-06-01T02:35:27.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:27.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]': finished 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: osdmap e23: 4 total, 3 up, 4 in 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 sudo[40034]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T02:35:27.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 sudo[40034]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 sudo[40034]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3882696569' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71b115be-d78f-4df3-89c6-15fb5639c913"}]': finished 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: osdmap e23: 4 total, 3 up, 4 in 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:27.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:27.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 sudo[27459]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-01T02:35:27.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 sudo[27459]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-06-01T02:35:27.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 sudo[27459]: pam_unix(sudo:session): session closed for user root 2024-06-01T02:35:28.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:28.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:28.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:27 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3559846365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3559846365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:27 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3559846365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:29.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:28 smithi146 ceph-mon[25945]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:29.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:28 smithi062 ceph-mon[25095]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:29.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:28 smithi062 ceph-mon[29843]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:31.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:30 smithi062 ceph-mon[25095]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:31.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:30 smithi062 ceph-mon[29843]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:31.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:30 smithi146 ceph-mon[25945]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:33.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:32 smithi146 ceph-mon[25945]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-06-01T02:35:33.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:32 smithi062 ceph-mon[25095]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-06-01T02:35:33.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:32 smithi062 ceph-mon[29843]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.1 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-06-01T02:35:34.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:35:34.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:34.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:33 smithi062 ceph-mon[25095]: Deploying daemon osd.3 on smithi062 2024-06-01T02:35:34.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:35:34.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:34.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:33 smithi062 ceph-mon[29843]: Deploying daemon osd.3 on smithi062 2024-06-01T02:35:34.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:35:34.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:34.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:33 smithi146 ceph-mon[25945]: Deploying daemon osd.3 on smithi062 2024-06-01T02:35:35.157 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:34 smithi062 ceph-mon[25095]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 101 KiB/s wr, 7 op/s 2024-06-01T02:35:35.157 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:34 smithi062 ceph-mon[29843]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 101 KiB/s wr, 7 op/s 2024-06-01T02:35:35.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:34 smithi146 ceph-mon[25945]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 101 KiB/s wr, 7 op/s 2024-06-01T02:35:37.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:36 smithi062 ceph-mon[25095]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 80 KiB/s wr, 5 op/s 2024-06-01T02:35:37.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:36 smithi062 ceph-mon[29843]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 80 KiB/s wr, 5 op/s 2024-06-01T02:35:37.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:36 smithi146 ceph-mon[25945]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 80 KiB/s wr, 5 op/s 2024-06-01T02:35:39.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[29843]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[25095]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:39.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:38 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:38 smithi146 ceph-mon[25945]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 67 KiB/s wr, 4 op/s 2024-06-01T02:35:39.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:38 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:39.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:38 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:39.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:38 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:39.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:38 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:41.079 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:40 smithi062 ceph-mon[25095]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-01T02:35:41.080 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:40 smithi062 ceph-mon[29843]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-01T02:35:41.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:40 smithi146 ceph-mon[25945]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-01T02:35:41.881 INFO:teuthology.orchestra.run.smithi062.stdout:Created osd(s) 3 on host 'smithi062' 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:42.205 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:42.206 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:42.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:35:42.498 DEBUG:teuthology.orchestra.run.smithi062:osd.3> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.3.service 2024-06-01T02:35:42.500 INFO:tasks.cephadm:Deploying osd.4 on smithi146 with /dev/vg_nvme/lv_4... 2024-06-01T02:35:42.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-01T02:35:43.379 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[25095]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T02:35:43.379 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[25095]: osdmap e24: 4 total, 3 up, 4 in 2024-06-01T02:35:43.379 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:43.379 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[25095]: from='osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:43.379 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[25095]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:43.380 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[29843]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T02:35:43.380 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[29843]: osdmap e24: 4 total, 3 up, 4 in 2024-06-01T02:35:43.380 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:43.380 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[29843]: from='osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:43.380 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-mon[29843]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:43.380 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[41479]: 2024-06-01T02:35:43.100+0000 7f391f21c700 -1 osd.3 0 waiting for initial osdmap 2024-06-01T02:35:43.380 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:35:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[41479]: 2024-06-01T02:35:43.105+0000 7f39193b2700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:35:43.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:43 smithi146 ceph-mon[25945]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T02:35:43.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:43 smithi146 ceph-mon[25945]: osdmap e24: 4 total, 3 up, 4 in 2024-06-01T02:35:43.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:43 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:43.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:43 smithi146 ceph-mon[25945]: from='osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:43.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:43 smithi146 ceph-mon[25945]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:35:44.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:44 smithi146 ceph-mon[25945]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:35:44.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:44 smithi146 ceph-mon[25945]: osdmap e25: 4 total, 3 up, 4 in 2024-06-01T02:35:44.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:44 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:44.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:44 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:44.226 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:44 smithi146 ceph-mon[25945]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[25095]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[25095]: osdmap e25: 4 total, 3 up, 4 in 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[25095]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[29843]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]': finished 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[29843]: osdmap e25: 4 total, 3 up, 4 in 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:44.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:44.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:44 smithi062 ceph-mon[29843]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-06-01T02:35:44.732 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-01T02:35:44.732 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-01T02:35:44.732 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-06-01T02:35:44.732 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-06-01T02:35:44.732 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113363 s, 925 MB/s 2024-06-01T02:35:44.732 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-06-01T02:35:45.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:45 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:35:45.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:45 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:35:45.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:45 smithi146 ceph-mon[25945]: osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246] boot 2024-06-01T02:35:45.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:45 smithi146 ceph-mon[25945]: osdmap e26: 4 total, 4 up, 4 in 2024-06-01T02:35:45.300 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:45 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:45.301 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_4 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[25095]: osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246] boot 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[25095]: osdmap e26: 4 total, 4 up, 4 in 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:35:45.448 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[29843]: osd.3 [v2:172.21.15.62:6826/2184160246,v1:172.21.15.62:6827/2184160246] boot 2024-06-01T02:35:45.449 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[29843]: osdmap e26: 4 total, 4 up, 4 in 2024-06-01T02:35:45.449 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:45 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: osdmap e27: 4 total, 4 up, 4 in 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi062 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi062 to 4016M 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:46 smithi146 ceph-mon[25945]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:46.511 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: osdmap e27: 4 total, 4 up, 4 in 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi062 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi062 to 4016M 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[29843]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: osdmap e27: 4 total, 4 up, 4 in 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi062 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi062 to 4016M 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:35:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:46 smithi062 ceph-mon[25095]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:47.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:47 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:35:47.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:47 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:35:47.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:47 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:47.761 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:47 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:35:47.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:47 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:35:47.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:47 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:47.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:47 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:35:47.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:47 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:35:47.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:47 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:48.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:48 smithi146 ceph-mon[25945]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:35:48.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:48 smithi146 ceph-mon[25945]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:48.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:48 smithi062 ceph-mon[29843]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:35:48.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:48 smithi062 ceph-mon[29843]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:48.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:48 smithi062 ceph-mon[25095]: from='client.24200 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:35:48.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:48 smithi062 ceph-mon[25095]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:50.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/2597483705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]: dispatch 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]: dispatch 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]': finished 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[25095]: osdmap e28: 5 total, 4 up, 5 in 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/2597483705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]: dispatch 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]: dispatch 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]': finished 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[29843]: osdmap e28: 5 total, 4 up, 5 in 2024-06-01T02:35:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:49 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:35:50.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:49 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/2597483705' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]: dispatch 2024-06-01T02:35:50.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:49 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]: dispatch 2024-06-01T02:35:50.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:49 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6009746c-ef5e-4635-9990-75af41fe143b"}]': finished 2024-06-01T02:35:50.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:49 smithi146 ceph-mon[25945]: osdmap e28: 5 total, 4 up, 5 in 2024-06-01T02:35:50.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:49 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:35:51.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:50 smithi062 ceph-mon[25095]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:51.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:50 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/3383284074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:51.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:50 smithi062 ceph-mon[29843]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:51.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:50 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/3383284074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:51.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:50 smithi146 ceph-mon[25945]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:51.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:50 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/3383284074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:35:52.784 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:52 smithi146 ceph-mon[25945]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:53.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:52 smithi062 ceph-mon[25095]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:53.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:52 smithi062 ceph-mon[29843]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:54.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:54 smithi146 ceph-mon[25945]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:55.011 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:54 smithi062 ceph-mon[25095]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:55.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:54 smithi062 ceph-mon[29843]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:55.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:55 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:35:55.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:55 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:56.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:35:56.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:55 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:56.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:35:56.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:55 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:35:56.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:56 smithi146 ceph-mon[25945]: Deploying daemon osd.4 on smithi146 2024-06-01T02:35:56.821 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:56 smithi146 ceph-mon[25945]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:57.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:56 smithi062 ceph-mon[25095]: Deploying daemon osd.4 on smithi146 2024-06-01T02:35:57.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:56 smithi062 ceph-mon[25095]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:57.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:56 smithi062 ceph-mon[29843]: Deploying daemon osd.4 on smithi146 2024-06-01T02:35:57.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:56 smithi062 ceph-mon[29843]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:58.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:35:58 smithi146 ceph-mon[25945]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:59.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:35:58 smithi062 ceph-mon[29843]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:35:59.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:35:58 smithi062 ceph-mon[25095]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:01.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:00 smithi146 ceph-mon[25945]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:01.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:01.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:01.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:01.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:00 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[25095]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[29843]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:01.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:01.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:01.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:00 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:03.066 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:02 smithi146 ceph-mon[25945]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:03.098 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 4 on host 'smithi146' 2024-06-01T02:36:03.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:02 smithi062 ceph-mon[25095]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:02 smithi062 ceph-mon[29843]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:03.668 DEBUG:teuthology.orchestra.run.smithi146:osd.4> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.4.service 2024-06-01T02:36:03.671 INFO:tasks.cephadm:Deploying osd.5 on smithi146 with /dev/vg_nvme/lv_3... 2024-06-01T02:36:03.671 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-01T02:36:04.288 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:04.288 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:04 smithi146 ceph-mon[25945]: from='osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:36:04.288 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:04 smithi146 ceph-mon[25945]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:36:04.288 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:04 smithi146 ceph-mon[25945]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:04.288 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:04 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:04.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[25095]: from='osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[25095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[25095]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[29843]: from='osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[29843]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[29843]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:04.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:04 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:05.396 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-mon[25945]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T02:36:05.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-mon[25945]: osdmap e29: 5 total, 4 up, 5 in 2024-06-01T02:36:05.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:05.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-mon[25945]: from='osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:05.397 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-mon[25945]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:05.397 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[29294]: 2024-06-01T02:36:05.119+0000 7fdd7e81b700 -1 osd.4 0 waiting for initial osdmap 2024-06-01T02:36:05.397 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:36:05 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[29294]: 2024-06-01T02:36:05.123+0000 7fdd791b2700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[25095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[25095]: osdmap e29: 5 total, 4 up, 5 in 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[25095]: from='osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[25095]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[29843]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[29843]: osdmap e29: 5 total, 4 up, 5 in 2024-06-01T02:36:05.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:05.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[29843]: from='osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:05.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:05 smithi062 ceph-mon[29843]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:06.328 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:06 smithi146 ceph-mon[25945]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:06.328 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:06 smithi146 ceph-mon[25945]: osdmap e30: 5 total, 4 up, 5 in 2024-06-01T02:36:06.328 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:06 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:06.328 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:06 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:06.328 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:06 smithi146 ceph-mon[25945]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:06.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[25095]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[25095]: osdmap e30: 5 total, 4 up, 5 in 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[25095]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[29843]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[29843]: osdmap e30: 5 total, 4 up, 5 in 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:06 smithi062 ceph-mon[29843]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-06-01T02:36:06.719 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-01T02:36:06.719 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-01T02:36:06.719 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-06-01T02:36:06.719 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-06-01T02:36:06.719 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125288 s, 837 MB/s 2024-06-01T02:36:06.719 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-06-01T02:36:07.364 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:07 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:36:07.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:07 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:36:07.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:07 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:07.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:07 smithi146 ceph-mon[25945]: osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247] boot 2024-06-01T02:36:07.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:07 smithi146 ceph-mon[25945]: osdmap e31: 5 total, 5 up, 5 in 2024-06-01T02:36:07.365 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:07 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:07.387 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_3 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[25095]: osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247] boot 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[25095]: osdmap e31: 5 total, 5 up, 5 in 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:07.413 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:36:07.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:36:07.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:07.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[29843]: osd.4 [v2:172.21.15.146:6800/1882199247,v1:172.21.15.146:6801/1882199247] boot 2024-06-01T02:36:07.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[29843]: osdmap e31: 5 total, 5 up, 5 in 2024-06-01T02:36:07.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:07 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:36:08.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:08 smithi062 ceph-mon[25095]: osdmap e32: 5 total, 5 up, 5 in 2024-06-01T02:36:08.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:08 smithi062 ceph-mon[25095]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:08.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:08 smithi062 ceph-mon[25095]: osdmap e33: 5 total, 5 up, 5 in 2024-06-01T02:36:08.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:08 smithi062 ceph-mon[29843]: osdmap e32: 5 total, 5 up, 5 in 2024-06-01T02:36:08.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:08 smithi062 ceph-mon[29843]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:08.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:08 smithi062 ceph-mon[29843]: osdmap e33: 5 total, 5 up, 5 in 2024-06-01T02:36:08.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:08 smithi146 ceph-mon[25945]: osdmap e32: 5 total, 5 up, 5 in 2024-06-01T02:36:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:08 smithi146 ceph-mon[25945]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:08.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:08 smithi146 ceph-mon[25945]: osdmap e33: 5 total, 5 up, 5 in 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: osdmap e34: 5 total, 5 up, 5 in 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi146 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi146 to 17088M 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:10.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: osdmap e34: 5 total, 5 up, 5 in 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi146 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi146 to 17088M 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:10.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:09 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: osdmap e34: 5 total, 5 up, 5 in 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi146 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi146 to 17088M 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:09 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:10.870 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: osdmap e35: 5 total, 5 up, 5 in 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:10.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: osdmap e35: 5 total, 5 up, 5 in 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:11.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: osdmap e35: 5 total, 5 up, 5 in 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: from='client.24227 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:11.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-06-01T02:36:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/1293970607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]: dispatch 2024-06-01T02:36:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]: dispatch 2024-06-01T02:36:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]': finished 2024-06-01T02:36:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: osdmap e36: 6 total, 5 up, 6 in 2024-06-01T02:36:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:12 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/2052403459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:36:13.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-06-01T02:36:13.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/1293970607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]: dispatch 2024-06-01T02:36:13.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]: dispatch 2024-06-01T02:36:13.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]': finished 2024-06-01T02:36:13.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: osdmap e36: 6 total, 5 up, 6 in 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/2052403459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/1293970607' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]: dispatch 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]: dispatch 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7b07195b-3822-4c75-829f-4e43ac03536b"}]': finished 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: osdmap e36: 6 total, 5 up, 6 in 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:12 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/2052403459' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:36:15.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:14 smithi146 ceph-mon[25945]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-06-01T02:36:15.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:14 smithi062 ceph-mon[25095]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-06-01T02:36:15.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:14 smithi062 ceph-mon[29843]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-06-01T02:36:16.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:16 smithi146 ceph-mon[25945]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-06-01T02:36:17.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:16 smithi062 ceph-mon[25095]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-06-01T02:36:17.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:16 smithi062 ceph-mon[29843]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-06-01T02:36:18.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:18 smithi146 ceph-mon[25945]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-01T02:36:18.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:18 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:36:18.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:18 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:19.102 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:18 smithi062 ceph-mon[25095]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-01T02:36:19.102 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:18 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:36:19.102 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:18 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:19.102 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:18 smithi062 ceph-mon[29843]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-06-01T02:36:19.102 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:18 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:36:19.102 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:18 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:20.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:19 smithi062 ceph-mon[25095]: Deploying daemon osd.5 on smithi146 2024-06-01T02:36:20.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:19 smithi062 ceph-mon[29843]: Deploying daemon osd.5 on smithi146 2024-06-01T02:36:20.083 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:19 smithi146 ceph-mon[25945]: Deploying daemon osd.5 on smithi146 2024-06-01T02:36:21.104 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:20 smithi146 ceph-mon[25945]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-01T02:36:21.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:20 smithi062 ceph-mon[25095]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-01T02:36:21.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:20 smithi062 ceph-mon[29843]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-01T02:36:23.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:22 smithi146 ceph-mon[25945]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:23.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:22 smithi062 ceph-mon[25095]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:23.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:22 smithi062 ceph-mon[29843]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:24.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:24.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:24.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:24.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:24 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:24.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:24 smithi146 ceph-mon[25945]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[29843]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:24.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:24.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:24.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:24 smithi062 ceph-mon[25095]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:26.786 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 5 on host 'smithi146' 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.142 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:27.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:27.143 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:27.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:27.473 DEBUG:teuthology.orchestra.run.smithi146:osd.5> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.5.service 2024-06-01T02:36:27.476 INFO:tasks.cephadm:Deploying osd.6 on smithi146 with /dev/vg_nvme/lv_2... 2024-06-01T02:36:27.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-01T02:36:28.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-mon[25945]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T02:36:28.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-mon[25945]: osdmap e37: 6 total, 5 up, 6 in 2024-06-01T02:36:28.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:28.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-mon[25945]: from='osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:28.023 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-mon[25945]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:28.307 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[32224]: 2024-06-01T02:36:28.023+0000 7f6c52fd3700 -1 osd.5 0 waiting for initial osdmap 2024-06-01T02:36:28.307 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:36:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[32224]: 2024-06-01T02:36:28.029+0000 7f6c4c167700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[25095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[25095]: osdmap e37: 6 total, 5 up, 6 in 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[25095]: from='osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[25095]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[29843]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[29843]: osdmap e37: 6 total, 5 up, 6 in 2024-06-01T02:36:28.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[29843]: from='osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:28 smithi062 ceph-mon[29843]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:29.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: osdmap e38: 6 total, 5 up, 6 in 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970] boot 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: osdmap e39: 6 total, 6 up, 6 in 2024-06-01T02:36:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:29 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: osdmap e38: 6 total, 5 up, 6 in 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970] boot 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: osdmap e39: 6 total, 6 up, 6 in 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-06-01T02:36:29.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: osdmap e38: 6 total, 5 up, 6 in 2024-06-01T02:36:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: osd.5 [v2:172.21.15.146:6808/2004940970,v1:172.21.15.146:6809/2004940970] boot 2024-06-01T02:36:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: osdmap e39: 6 total, 6 up, 6 in 2024-06-01T02:36:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:29 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:36:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:30 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:36:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:30 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:36:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:30 smithi146 ceph-mon[25945]: osdmap e40: 6 total, 6 up, 6 in 2024-06-01T02:36:30.482 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-01T02:36:30.482 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-01T02:36:30.482 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-06-01T02:36:30.482 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-06-01T02:36:30.482 INFO:teuthology.orchestra.run.smithi146.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123223 s, 851 MB/s 2024-06-01T02:36:30.482 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-06-01T02:36:30.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:30 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:36:30.511 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:30 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:36:30.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:30 smithi062 ceph-mon[25095]: osdmap e40: 6 total, 6 up, 6 in 2024-06-01T02:36:30.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:30 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:36:30.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:30 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:36:30.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:30 smithi062 ceph-mon[29843]: osdmap e40: 6 total, 6 up, 6 in 2024-06-01T02:36:31.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_2 2024-06-01T02:36:31.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:31 smithi146 ceph-mon[25945]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:31.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:31 smithi146 ceph-mon[25945]: osdmap e41: 6 total, 6 up, 6 in 2024-06-01T02:36:31.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:31 smithi062 ceph-mon[25095]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:31.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:31 smithi062 ceph-mon[25095]: osdmap e41: 6 total, 6 up, 6 in 2024-06-01T02:36:31.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:31 smithi062 ceph-mon[29843]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:31.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:31 smithi062 ceph-mon[29843]: osdmap e41: 6 total, 6 up, 6 in 2024-06-01T02:36:32.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi146 2024-06-01T02:36:32.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:32.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:32.906 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:32.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi146 to 8544M 2024-06-01T02:36:32.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:32.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:32.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:32.907 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:32 smithi146 ceph-mon[25945]: osdmap e42: 6 total, 6 up, 6 in 2024-06-01T02:36:33.011 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi146 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi146 to 8544M 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[29843]: osdmap e42: 6 total, 6 up, 6 in 2024-06-01T02:36:33.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi146 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi146 to 8544M 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: pgmap v132: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:32 smithi062 ceph-mon[25095]: osdmap e42: 6 total, 6 up, 6 in 2024-06-01T02:36:34.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:33 smithi146 ceph-mon[25945]: osdmap e43: 6 total, 6 up, 6 in 2024-06-01T02:36:34.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:34.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:34.097 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:33 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[25095]: osdmap e43: 6 total, 6 up, 6 in 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[29843]: osdmap e43: 6 total, 6 up, 6 in 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:34.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:34.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:33 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:34.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:34 smithi146 ceph-mon[25945]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:34.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:34 smithi146 ceph-mon[25945]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:35.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:34 smithi062 ceph-mon[25095]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:35.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:34 smithi062 ceph-mon[25095]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:35.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:34 smithi062 ceph-mon[29843]: from='client.24254 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:35.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:34 smithi062 ceph-mon[29843]: pgmap v135: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:35 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/141262432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]: dispatch 2024-06-01T02:36:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:35 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]: dispatch 2024-06-01T02:36:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:35 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]': finished 2024-06-01T02:36:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:35 smithi146 ceph-mon[25945]: osdmap e44: 7 total, 6 up, 7 in 2024-06-01T02:36:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:35 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/141262432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]: dispatch 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]: dispatch 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]': finished 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[25095]: osdmap e44: 7 total, 6 up, 7 in 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/141262432' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]: dispatch 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]: dispatch 2024-06-01T02:36:36.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3fff939-031b-4431-9518-b90185765c3b"}]': finished 2024-06-01T02:36:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[29843]: osdmap e44: 7 total, 6 up, 7 in 2024-06-01T02:36:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:35 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:37.163 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:36 smithi062 ceph-mon[29843]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:37.163 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:36 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/2231473390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:36:37.163 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:36 smithi062 ceph-mon[25095]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:37.163 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:36 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/2231473390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:36:37.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:36 smithi146 ceph-mon[25945]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:37.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:36 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/2231473390' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:36:39.050 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:38 smithi146 ceph-mon[25945]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:39.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:38 smithi062 ceph-mon[25095]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:39.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:38 smithi062 ceph-mon[29843]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:41.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:40 smithi146 ceph-mon[25945]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:41.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:40 smithi062 ceph-mon[25095]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:41.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:40 smithi062 ceph-mon[29843]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:42.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:42 smithi146 ceph-mon[25945]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:42.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:36:42.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:42 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:42.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:42 smithi146 ceph-mon[25945]: Deploying daemon osd.6 on smithi146 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[25095]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[25095]: Deploying daemon osd.6 on smithi146 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[29843]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:43.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:42 smithi062 ceph-mon[29843]: Deploying daemon osd.6 on smithi146 2024-06-01T02:36:44.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:44 smithi146 ceph-mon[25945]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:45.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:44 smithi062 ceph-mon[25095]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:45.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:44 smithi062 ceph-mon[29843]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:46.897 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:46 smithi146 ceph-mon[25945]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:47.185 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:46 smithi062 ceph-mon[25095]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:47.185 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:46 smithi062 ceph-mon[29843]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:48 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:48 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:48 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:48 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:48 smithi146 ceph-mon[25945]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[29843]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:48 smithi062 ceph-mon[25095]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:50.701 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 6 on host 'smithi146' 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:50.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:50.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:50.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:50 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:36:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:36:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:36:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:51.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:50 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:36:51.304 DEBUG:teuthology.orchestra.run.smithi146:osd.6> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.6.service 2024-06-01T02:36:51.306 INFO:tasks.cephadm:Deploying osd.7 on smithi146 with /dev/vg_nvme/lv_1... 2024-06-01T02:36:51.306 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-06-01T02:36:52.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-mon[25945]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-01T02:36:52.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-mon[25945]: osdmap e45: 7 total, 6 up, 7 in 2024-06-01T02:36:52.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:52.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-mon[25945]: from='osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:52.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-mon[25945]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:52.205 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[35104]: 2024-06-01T02:36:51.894+0000 7fe3388cb700 -1 osd.6 0 waiting for initial osdmap 2024-06-01T02:36:52.205 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:36:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[35104]: 2024-06-01T02:36:51.900+0000 7fe333262700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:36:52.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[25095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-01T02:36:52.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[25095]: osdmap e45: 7 total, 6 up, 7 in 2024-06-01T02:36:52.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:52.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[25095]: from='osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:52.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[25095]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:52.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[29843]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-01T02:36:52.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[29843]: osdmap e45: 7 total, 6 up, 7 in 2024-06-01T02:36:52.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:52.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[29843]: from='osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:52.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:51 smithi062 ceph-mon[29843]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:36:52.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:52 smithi146 ceph-mon[25945]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:52.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:52 smithi146 ceph-mon[25945]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:52.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:52 smithi146 ceph-mon[25945]: osdmap e46: 7 total, 6 up, 7 in 2024-06-01T02:36:52.993 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:52 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:52.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:52 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[25095]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[25095]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[25095]: osdmap e46: 7 total, 6 up, 7 in 2024-06-01T02:36:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[29843]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-06-01T02:36:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[29843]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:36:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[29843]: osdmap e46: 7 total, 6 up, 7 in 2024-06-01T02:36:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:52 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:54.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:53 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:36:54.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:53 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:36:54.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:53 smithi146 ceph-mon[25945]: osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728] boot 2024-06-01T02:36:54.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:53 smithi146 ceph-mon[25945]: osdmap e47: 7 total, 7 up, 7 in 2024-06-01T02:36:54.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:53 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:54.055 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:53 smithi146 ceph-mon[25945]: osdmap e48: 7 total, 7 up, 7 in 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[25095]: osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728] boot 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[25095]: osdmap e47: 7 total, 7 up, 7 in 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[25095]: osdmap e48: 7 total, 7 up, 7 in 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:36:54.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:36:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[29843]: osd.6 [v2:172.21.15.146:6816/3392045728,v1:172.21.15.146:6817/3392045728] boot 2024-06-01T02:36:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[29843]: osdmap e47: 7 total, 7 up, 7 in 2024-06-01T02:36:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:36:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:53 smithi062 ceph-mon[29843]: osdmap e48: 7 total, 7 up, 7 in 2024-06-01T02:36:54.954 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:54 smithi146 ceph-mon[25945]: osdmap e49: 7 total, 7 up, 7 in 2024-06-01T02:36:55.178 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-06-01T02:36:55.179 INFO:teuthology.orchestra.run.smithi146.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-06-01T02:36:55.179 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10+0 records in 2024-06-01T02:36:55.179 INFO:teuthology.orchestra.run.smithi146.stderr:10+0 records out 2024-06-01T02:36:55.179 INFO:teuthology.orchestra.run.smithi146.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113505 s, 924 MB/s 2024-06-01T02:36:55.179 INFO:teuthology.orchestra.run.smithi146.stderr:--> Zapping successful for: 2024-06-01T02:36:55.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:55.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:55.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:55.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:55.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[25095]: osdmap e49: 7 total, 7 up, 7 in 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:36:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:54 smithi062 ceph-mon[29843]: osdmap e49: 7 total, 7 up, 7 in 2024-06-01T02:36:55.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi146:vg_nvme/lv_1 2024-06-01T02:36:56.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:55 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi146 2024-06-01T02:36:56.068 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:55 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi146 to 5696M 2024-06-01T02:36:56.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:55 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi146 2024-06-01T02:36:56.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:55 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi146 to 5696M 2024-06-01T02:36:56.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:55 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi146 2024-06-01T02:36:56.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:55 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi146 to 5696M 2024-06-01T02:36:56.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:56 smithi146 ceph-mon[25945]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:56.981 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:56 smithi146 ceph-mon[25945]: osdmap e50: 7 total, 7 up, 7 in 2024-06-01T02:36:57.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:56 smithi062 ceph-mon[25095]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:57.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:56 smithi062 ceph-mon[25095]: osdmap e50: 7 total, 7 up, 7 in 2024-06-01T02:36:57.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:56 smithi062 ceph-mon[29843]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:57.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:56 smithi062 ceph-mon[29843]: osdmap e50: 7 total, 7 up, 7 in 2024-06-01T02:36:58.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:57 smithi146 ceph-mon[25945]: osdmap e51: 7 total, 7 up, 7 in 2024-06-01T02:36:58.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:58.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:58.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:57 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:58.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[25095]: osdmap e51: 7 total, 7 up, 7 in 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[29843]: osdmap e51: 7 total, 7 up, 7 in 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-01T02:36:58.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:57 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:36:58.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:58 smithi146 ceph-mon[25945]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:58.949 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:58 smithi146 ceph-mon[25945]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:59.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:58 smithi062 ceph-mon[25095]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:59.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:58 smithi062 ceph-mon[25095]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:36:59.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:58 smithi062 ceph-mon[29843]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi146:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:36:59.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:58 smithi062 ceph-mon[29843]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/3967756626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]: dispatch 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]: dispatch 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[25095]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]': finished 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[25095]: osdmap e52: 8 total, 7 up, 8 in 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/3967756626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]: dispatch 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]: dispatch 2024-06-01T02:37:00.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[29843]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]': finished 2024-06-01T02:37:00.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[29843]: osdmap e52: 8 total, 7 up, 8 in 2024-06-01T02:37:00.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:36:59 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:00.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:59 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/3967756626' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]: dispatch 2024-06-01T02:37:00.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:59 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]: dispatch 2024-06-01T02:37:00.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:59 smithi146 ceph-mon[25945]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "06ee51a3-2d2b-4636-b660-a74b6aeaafb8"}]': finished 2024-06-01T02:37:00.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:59 smithi146 ceph-mon[25945]: osdmap e52: 8 total, 7 up, 8 in 2024-06-01T02:37:00.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:36:59 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:00 smithi062 ceph-mon[25095]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-01T02:37:01.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:00 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/1507387351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:37:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:00 smithi062 ceph-mon[29843]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-01T02:37:01.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:00 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/1507387351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:37:01.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:00 smithi146 ceph-mon[25945]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-01T02:37:01.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:00 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/1507387351' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-01T02:37:03.031 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:02 smithi146 ceph-mon[25945]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-01T02:37:03.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:02 smithi062 ceph-mon[25095]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-01T02:37:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:02 smithi062 ceph-mon[29843]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-06-01T02:37:05.140 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:04 smithi146 ceph-mon[25945]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-06-01T02:37:05.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:04 smithi062 ceph-mon[25095]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-06-01T02:37:05.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:04 smithi062 ceph-mon[29843]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-06-01T02:37:07.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:06 smithi062 ceph-mon[25095]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-01T02:37:07.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:37:07.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:06 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:07.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:07 smithi062 ceph-mon[29843]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-01T02:37:07.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:07 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:37:07.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:07 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:07.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:07 smithi146 ceph-mon[25945]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-01T02:37:07.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:07 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:37:07.275 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:07 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:08.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:08 smithi146 ceph-mon[25945]: Deploying daemon osd.7 on smithi146 2024-06-01T02:37:08.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:08 smithi062 ceph-mon[25095]: Deploying daemon osd.7 on smithi146 2024-06-01T02:37:08.261 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:08 smithi062 ceph-mon[29843]: Deploying daemon osd.7 on smithi146 2024-06-01T02:37:09.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:09 smithi146 ceph-mon[25945]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-01T02:37:09.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:09 smithi062 ceph-mon[25095]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-01T02:37:09.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:09 smithi062 ceph-mon[29843]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-06-01T02:37:10.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:10.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:10 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:10.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:10.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:10 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:10.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:10 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:11.175 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:11 smithi146 ceph-mon[25945]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:11.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:11 smithi062 ceph-mon[25095]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:11.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:11 smithi062 ceph-mon[29843]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:12.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:12.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:12.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:12.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:12.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[29843]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:12.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:12.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:12.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:12.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:12.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:12 smithi062 ceph-mon[25095]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:12.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:12.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:12.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:12.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:12 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:12.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:12 smithi146 ceph-mon[25945]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:14.994 INFO:teuthology.orchestra.run.smithi146.stdout:Created osd(s) 7 on host 'smithi146' 2024-06-01T02:37:15.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:14 smithi146 ceph-mon[25945]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:15.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:14 smithi146 ceph-mon[25945]: from='osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:37:15.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:14 smithi146 ceph-mon[25945]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:37:15.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:14 smithi062 ceph-mon[29843]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:15.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:14 smithi062 ceph-mon[29843]: from='osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:37:15.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:14 smithi062 ceph-mon[29843]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:37:15.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:14 smithi062 ceph-mon[25095]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:15.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:14 smithi062 ceph-mon[25095]: from='osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:37:15.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:14 smithi062 ceph-mon[25095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:37:15.663 DEBUG:teuthology.orchestra.run.smithi146:osd.7> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.7.service 2024-06-01T02:37:15.666 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-06-01T02:37:15.666 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-01T02:37:15.955 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[38004]: 2024-06-01T02:37:15.840+0000 7fd01d6d3700 -1 osd.7 0 waiting for initial osdmap 2024-06-01T02:37:15.955 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[38004]: 2024-06-01T02:37:15.847+0000 7fd01a06e700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: osdmap e53: 8 total, 7 up, 8 in 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: from='osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:15.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:15 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: osdmap e53: 8 total, 7 up, 8 in 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: from='osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-01T02:37:15.992 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: osdmap e53: 8 total, 7 up, 8 in 2024-06-01T02:37:15.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:15.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: from='osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:37:15.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:37:15.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:15.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:15 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:16.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:16 smithi146 ceph-mon[25945]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:16.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:16 smithi146 ceph-mon[25945]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:37:16.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:16 smithi146 ceph-mon[25945]: osdmap e54: 8 total, 7 up, 8 in 2024-06-01T02:37:16.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:16 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:16.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:16 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:17.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[25095]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:17.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[25095]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:37:17.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[25095]: osdmap e54: 8 total, 7 up, 8 in 2024-06-01T02:37:17.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:17.117 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:17.117 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[29843]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-06-01T02:37:17.118 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[29843]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]': finished 2024-06-01T02:37:17.118 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[29843]: osdmap e54: 8 total, 7 up, 8 in 2024-06-01T02:37:17.118 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:17.118 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:16 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:17.774 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:37:17.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: purged_snaps scrub starts 2024-06-01T02:37:17.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: purged_snaps scrub ok 2024-06-01T02:37:17.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056] boot 2024-06-01T02:37:17.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: osdmap e55: 8 total, 8 up, 8 in 2024-06-01T02:37:17.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:17.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/564000415' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T02:37:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:17 smithi146 ceph-mon[25945]: osdmap e56: 8 total, 8 up, 8 in 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: purged_snaps scrub starts 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: purged_snaps scrub ok 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056] boot 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: osdmap e55: 8 total, 8 up, 8 in 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/564000415' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[25095]: osdmap e56: 8 total, 8 up, 8 in 2024-06-01T02:37:18.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: purged_snaps scrub starts 2024-06-01T02:37:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: purged_snaps scrub ok 2024-06-01T02:37:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: osd.7 [v2:172.21.15.146:6824/2912039056,v1:172.21.15.146:6825/2912039056] boot 2024-06-01T02:37:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: osdmap e55: 8 total, 8 up, 8 in 2024-06-01T02:37:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/564000415' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-01T02:37:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:17 smithi062 ceph-mon[29843]: osdmap e56: 8 total, 8 up, 8 in 2024-06-01T02:37:18.533 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1717209436,"num_in_osds":8,"osd_in_since":1717209419,"num_remapped_pgs":1} 2024-06-01T02:37:18.533 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-01T02:37:18.775 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:37:18.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:18 smithi146 ceph-mon[25945]: pgmap v169: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:18.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:18 smithi146 ceph-mon[25945]: osdmap e57: 8 total, 8 up, 8 in 2024-06-01T02:37:19.123 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:18 smithi062 ceph-mon[25095]: pgmap v169: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:19.123 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:18 smithi062 ceph-mon[25095]: osdmap e57: 8 total, 8 up, 8 in 2024-06-01T02:37:19.123 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:18 smithi062 ceph-mon[29843]: pgmap v169: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:19.123 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:18 smithi062 ceph-mon[29843]: osdmap e57: 8 total, 8 up, 8 in 2024-06-01T02:37:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: Detected new or changed devices on smithi146 2024-06-01T02:37:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:19 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.223 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: Detected new or changed devices on smithi146 2024-06-01T02:37:20.223 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: Detected new or changed devices on smithi146 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.224 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:20.225 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:19 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:20.654 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:37:20.654 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":58,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","created":"2024-06-01T02:32:14.312516+0000","modified":"2024-06-01T02:37:20.084553+0000","last_up_change":"2024-06-01T02:37:16.834197+0000","last_in_change":"2024-06-01T02:36:59.938186+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-01T02:35:20.229606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0e376b0d-882e-48c6-b47f-8d68368ff47b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6803","nonce":2961277760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6805","nonce":2961277760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6809","nonce":2961277760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6807","nonce":2961277760}]},"public_addr":"172.21.15.62:6803/2961277760","cluster_addr":"172.21.15.62:6805/2961277760","heartbeat_back_addr":"172.21.15.62:6809/2961277760","heartbeat_front_addr":"172.21.15.62:6807/2961277760","state":["exists","up"]},{"osd":1,"uuid":"3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6811","nonce":42392874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6813","nonce":42392874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6817","nonce":42392874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6815","nonce":42392874}]},"public_addr":"172.21.15.62:6811/42392874","cluster_addr":"172.21.15.62:6813/42392874","heartbeat_back_addr":"172.21.15.62:6817/42392874","heartbeat_front_addr":"172.21.15.62:6815/42392874","state":["exists","up"]},{"osd":2,"uuid":"c16629b6-e552-407a-b74b-d2beb921a2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6819","nonce":3316357651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6821","nonce":3316357651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6825","nonce":3316357651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6823","nonce":3316357651}]},"public_addr":"172.21.15.62:6819/3316357651","cluster_addr":"172.21.15.62:6821/3316357651","heartbeat_back_addr":"172.21.15.62:6825/3316357651","heartbeat_front_addr":"172.21.15.62:6823/3316357651","state":["exists","up"]},{"osd":3,"uuid":"71b115be-d78f-4df3-89c6-15fb5639c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6827","nonce":2184160246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6829","nonce":2184160246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6833","nonce":2184160246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6831","nonce":2184160246}]},"public_addr":"172.21.15.62:6827/2184160246","cluster_addr":"172.21.15.62:6829/2184160246","heartbeat_back_addr":"172.21.15.62:6833/2184160246","heartbeat_front_addr":"172.21.15.62:6831/2184160246","state":["exists","up"]},{"osd":4,"uuid":"6009746c-ef5e-4635-9990-75af41fe143b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6801","nonce":1882199247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6803","nonce":1882199247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6807","nonce":1882199247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6805","nonce":1882199247}]},"public_addr":"172.21.15.146:6801/1882199247","cluster_addr":"172.21.15.146:6803/1882199247","heartbeat_back_addr":"172.21.15.146:6807/1882199247","heartbeat_front_addr":"172.21.15.146:6805/1882199247","state":["exists","up"]},{"osd":5,"uuid":"7b07195b-3822-4c75-829f-4e43ac03536b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6809","nonce":2004940970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6811","nonce":2004940970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6815","nonce":2004940970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6813","nonce":2004940970}]},"public_addr":"172.21.15.146:6809/2004940970","cluster_addr":"172.21.15.146:6811/2004940970","heartbeat_back_addr":"172.21.15.146:6815/2004940970","heartbeat_front_addr":"172.21.15.146:6813/2004940970","state":["exists","up"]},{"osd":6,"uuid":"c3fff939-031b-4431-9518-b90185765c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6817","nonce":3392045728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6819","nonce":3392045728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6823","nonce":3392045728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6821","nonce":3392045728}]},"public_addr":"172.21.15.146:6817/3392045728","cluster_addr":"172.21.15.146:6819/3392045728","heartbeat_back_addr":"172.21.15.146:6823/3392045728","heartbeat_front_addr":"172.21.15.146:6821/3392045728","state":["exists","up"]},{"osd":7,"uuid":"06ee51a3-2d2b-4636-b660-a74b6aeaafb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6825","nonce":2912039056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6827","nonce":2912039056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6831","nonce":2912039056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6829","nonce":2912039056}]},"public_addr":"172.21.15.146:6825/2912039056","cluster_addr":"172.21.15.146:6827/2912039056","heartbeat_back_addr":"172.21.15.146:6831/2912039056","heartbeat_front_addr":"172.21.15.146:6829/2912039056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:34:28.509822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:34:52.874848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:35:16.346228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:35:42.435488+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:04.277597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:27.522336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:51.282685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:37:15.589252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6801/3339555501":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/4106139753":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3491871":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/1842014486":"2024-06-02T02:33:09.305931+0000","172.21.15.62:6800/1434776916":"2024-06-02T02:32:46.207475+0000","172.21.15.62:0/1726234481":"2024-06-02T02:32:46.207475+0000","172.21.15.62:6801/1434776916":"2024-06-02T02:32:46.207475+0000","172.21.15.62:0/2403150514":"2024-06-02T02:32:46.207475+0000","172.21.15.62:6800/3339555501":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3889207289":"2024-06-02T02:32:46.207475+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-01T02:37:21.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi146 to 4272M 2024-06-01T02:37:21.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[25095]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:21.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[25095]: osdmap e58: 8 total, 8 up, 8 in 2024-06-01T02:37:21.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4133194879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:21.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi146 to 4272M 2024-06-01T02:37:21.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[29843]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:21.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[29843]: osdmap e58: 8 total, 8 up, 8 in 2024-06-01T02:37:21.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:20 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4133194879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:21.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:20 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi146 to 4272M 2024-06-01T02:37:21.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:20 smithi146 ceph-mon[25945]: pgmap v172: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:21.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:20 smithi146 ceph-mon[25945]: osdmap e58: 8 total, 8 up, 8 in 2024-06-01T02:37:21.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:20 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4133194879' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:21.363 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-01T02:35:20.229606+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-01T02:37:21.363 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-01T02:37:21.587 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:37:22.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:22 smithi062 ceph-mon[25095]: osdmap e59: 8 total, 8 up, 8 in 2024-06-01T02:37:22.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:22 smithi062 ceph-mon[25095]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:22.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:22 smithi062 ceph-mon[29843]: osdmap e59: 8 total, 8 up, 8 in 2024-06-01T02:37:22.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:22 smithi062 ceph-mon[29843]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:22.454 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:22 smithi146 ceph-mon[25945]: osdmap e59: 8 total, 8 up, 8 in 2024-06-01T02:37:22.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:22 smithi146 ceph-mon[25945]: pgmap v175: 1 pgs: 1 peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:23.211 INFO:teuthology.orchestra.run.smithi062.stdout:pg_num: 1 2024-06-01T02:37:23.704 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:23 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2942221179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-01T02:37:23.708 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:23 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2942221179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-01T02:37:23.708 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:23 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2942221179' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-01T02:37:23.733 INFO:tasks.cephadm:Adding prometheus.a on smithi146 2024-06-01T02:37:23.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch apply prometheus '1;smithi146=a' 2024-06-01T02:37:25.047 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:24 smithi146 ceph-mon[25945]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-06-01T02:37:25.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:24 smithi062 ceph-mon[25095]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-06-01T02:37:25.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:24 smithi062 ceph-mon[29843]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-06-01T02:37:25.710 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled prometheus update... 2024-06-01T02:37:26.312 DEBUG:teuthology.orchestra.run.smithi146:prometheus.a> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a.service 2024-06-01T02:37:26.314 INFO:tasks.cephadm:Adding node-exporter.a on smithi062 2024-06-01T02:37:26.314 INFO:tasks.cephadm:Adding node-exporter.b on smithi146 2024-06-01T02:37:26.315 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch apply node-exporter '2;smithi062=a;smithi146=b' 2024-06-01T02:37:26.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ignoring --setuser ceph since I am not root 2024-06-01T02:37:26.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ignoring --setgroup ceph since I am not root 2024-06-01T02:37:26.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mgr[27124]: -- 172.21.15.146:0/4033671588 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5650b3bb2340 con 0x5650b3b8b000 2024-06-01T02:37:26.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:26.900+0000 7f935fe0d000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:37:26.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: Saving service prometheus spec with placement smithi146=a;count:1 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-01T02:37:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:26 smithi146 ceph-mon[25945]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-01T02:37:27.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setuser ceph since I am not root 2024-06-01T02:37:27.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setgroup ceph since I am not root 2024-06-01T02:37:27.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mgr[25324]: -- 172.21.15.62:0/3281877578 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e088074340 con 0x55e08804cc00 2024-06-01T02:37:27.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:26.917+0000 7f7641a5e000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:37:27.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:27.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: Saving service prometheus spec with placement smithi146=a;count:1 2024-06-01T02:37:27.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:27.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:27.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:27.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:27.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:27.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-01T02:37:27.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[25095]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-01T02:37:27.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='client.24326 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:27.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: Saving service prometheus spec with placement smithi146=a;count:1 2024-06-01T02:37:27.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:27.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:27.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:27.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:27.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' 2024-06-01T02:37:27.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-01T02:37:27.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:26 smithi062 ceph-mon[29843]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-06-01T02:37:27.354 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:27.038+0000 7f935fe0d000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:37:27.354 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:27.135+0000 7f935fe0d000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:37:27.473 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:27.053+0000 7f7641a5e000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:37:27.473 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:27.154+0000 7f7641a5e000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:37:27.473 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:27.474+0000 7f7641a5e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:37:27.670 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:27.440+0000 7f935fe0d000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:37:27.671 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:27.521+0000 7f935fe0d000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:37:27.739 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:27.560+0000 7f7641a5e000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:37:27.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:27 smithi146 ceph-mon[25945]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-01T02:37:27.935 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:27 smithi146 ceph-mon[25945]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-01T02:37:28.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-mon[25095]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-01T02:37:28.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-mon[25095]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-01T02:37:28.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-mon[29843]: from='mgr.14152 172.21.15.62:0/673448636' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-01T02:37:28.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:27 smithi062 ceph-mon[29843]: mgrmap e16: y(active, since 4m), standbys: x 2024-06-01T02:37:28.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:28.661+0000 7f935fe0d000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:37:28.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:28.825+0000 7f935fe0d000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:37:28.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:28.906+0000 7f935fe0d000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:37:29.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:28 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:28.745+0000 7f7641a5e000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:37:29.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:28 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:28.918+0000 7f7641a5e000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:37:29.346 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:29.017+0000 7f7641a5e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:37:29.633 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:29.346+0000 7f7641a5e000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:37:29.633 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:29.441+0000 7f7641a5e000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:37:29.633 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:29.538+0000 7f7641a5e000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:37:29.640 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:29.218+0000 7f935fe0d000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:37:29.640 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:29.308+0000 7f935fe0d000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:37:29.641 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:29.400+0000 7f935fe0d000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:37:29.641 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:29.641+0000 7f935fe0d000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:37:30.011 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:29 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:29.814+0000 7f7641a5e000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:37:30.454 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:30.080+0000 7f935fe0d000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:37:30.761 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:30.278+0000 7f7641a5e000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:37:30.954 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:30.586+0000 7f935fe0d000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:37:30.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:30.679+0000 7f935fe0d000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:37:31.194 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:30.787+0000 7f7641a5e000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:37:31.194 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:30.885+0000 7f7641a5e000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:37:31.194 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:31 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:31.194+0000 7f7641a5e000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:37:31.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:30.970+0000 7f935fe0d000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:37:31.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:31.059+0000 7f935fe0d000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:37:31.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:31 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:31.288+0000 7f7641a5e000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:37:31.795 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:31.461+0000 7f935fe0d000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:37:31.795 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:31.591+0000 7f935fe0d000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:37:31.796 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:31.682+0000 7f935fe0d000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:37:31.796 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:31.796+0000 7f935fe0d000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:37:32.011 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:31 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:31.716+0000 7f7641a5e000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:37:32.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:31 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:31.855+0000 7f7641a5e000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:37:32.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:31 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:31.952+0000 7f7641a5e000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:37:32.204 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:31.884+0000 7f935fe0d000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:37:32.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:32.072+0000 7f7641a5e000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:37:32.512 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:32.165+0000 7f7641a5e000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:37:32.704 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:32.428+0000 7f935fe0d000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:32.743+0000 7f7641a5e000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[29843]: Standby manager daemon x restarted 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[29843]: Standby manager daemon x started 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:37:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:37:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[25095]: Standby manager daemon x restarted 2024-06-01T02:37:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[25095]: Standby manager daemon x started 2024-06-01T02:37:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:37:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:37:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:37:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:32 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:37:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-mon[25945]: Standby manager daemon x restarted 2024-06-01T02:37:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-mon[25945]: Standby manager daemon x started 2024-06-01T02:37:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:37:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:37:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:37:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3844008985' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:32.787+0000 7f935fe0d000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: [01/Jun/2024:02:37:32] ENGINE Bus STARTING 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: CherryPy Checker: 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: The Application mounted at '' has an empty config. 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: [01/Jun/2024:02:37:32] ENGINE Serving on http://:::9283 2024-06-01T02:37:33.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: [01/Jun/2024:02:37:32] ENGINE Bus STARTED 2024-06-01T02:37:33.511 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:33.121+0000 7f7641a5e000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:37:34.176 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[25095]: mgrmap e17: y(active, since 4m), standbys: x 2024-06-01T02:37:34.176 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[25095]: Active manager daemon y restarted 2024-06-01T02:37:34.176 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[25095]: Activating manager daemon y 2024-06-01T02:37:34.176 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[25095]: osdmap e60: 8 total, 8 up, 8 in 2024-06-01T02:37:34.176 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[29843]: mgrmap e17: y(active, since 4m), standbys: x 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[29843]: Active manager daemon y restarted 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[29843]: Activating manager daemon y 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-mon[29843]: osdmap e60: 8 total, 8 up, 8 in 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:33 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:37:33] ENGINE Bus STARTING 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: CherryPy Checker: 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: The Application mounted at '' has an empty config. 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:37:34] ENGINE Serving on http://:::9283 2024-06-01T02:37:34.177 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:37:34] ENGINE Bus STARTED 2024-06-01T02:37:34.180 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:33 smithi146 ceph-mon[25945]: mgrmap e17: y(active, since 4m), standbys: x 2024-06-01T02:37:34.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:33 smithi146 ceph-mon[25945]: Active manager daemon y restarted 2024-06-01T02:37:34.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:33 smithi146 ceph-mon[25945]: Activating manager daemon y 2024-06-01T02:37:34.181 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:33 smithi146 ceph-mon[25945]: osdmap e60: 8 total, 8 up, 8 in 2024-06-01T02:37:34.882 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled node-exporter update... 2024-06-01T02:37:34.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: mgrmap e18: y(active, starting, since 0.740024s), standbys: x 2024-06-01T02:37:34.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:37:34.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:37:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: Manager daemon y is now available 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:34.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:35.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: mgrmap e18: y(active, starting, since 0.740024s), standbys: x 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:37:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: Manager daemon y is now available 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:35.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: mgrmap e18: y(active, starting, since 0.740024s), standbys: x 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:37:35.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: Manager daemon y is now available 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:35.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:37:35.648 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:37:35] ENGINE Bus STARTING 2024-06-01T02:37:35.648 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:37:35] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:37:35.649 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:37:35] ENGINE Bus STARTED 2024-06-01T02:37:35.728 DEBUG:teuthology.orchestra.run.smithi062:node-exporter.a> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service 2024-06-01T02:37:35.731 DEBUG:teuthology.orchestra.run.smithi146:node-exporter.b> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service 2024-06-01T02:37:35.734 INFO:tasks.cephadm:Adding alertmanager.a on smithi062 2024-06-01T02:37:35.734 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch apply alertmanager '1;smithi062=a' 2024-06-01T02:37:35.928 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: mgrmap e19: y(active, since 1.74635s), standbys: x 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: Saving service node-exporter spec with placement smithi062=a;smithi146=b;count:2 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: [01/Jun/2024:02:37:35] ENGINE Bus STARTING 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: [01/Jun/2024:02:37:35] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: [01/Jun/2024:02:37:35] ENGINE Bus STARTED 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: mgrmap e19: y(active, since 1.74635s), standbys: x 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: Saving service node-exporter spec with placement smithi062=a;smithi146=b;count:2 2024-06-01T02:37:35.929 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: [01/Jun/2024:02:37:35] ENGINE Bus STARTING 2024-06-01T02:37:35.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: [01/Jun/2024:02:37:35] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:37:35.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: [01/Jun/2024:02:37:35] ENGINE Bus STARTED 2024-06-01T02:37:35.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: mgrmap e19: y(active, since 1.74635s), standbys: x 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: Saving service node-exporter spec with placement smithi062=a;smithi146=b;count:2 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: [01/Jun/2024:02:37:35] ENGINE Bus STARTING 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: [01/Jun/2024:02:37:35] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: [01/Jun/2024:02:37:35] ENGINE Bus STARTED 2024-06-01T02:37:35.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: mgrmap e20: y(active, since 2s), standbys: x 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:37:37.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:36 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:37:37.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:37.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: mgrmap e20: y(active, since 2s), standbys: x 2024-06-01T02:37:37.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:37:37.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: mgrmap e20: y(active, since 2s), standbys: x 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:37:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:37.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:37:37.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:36 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:37:37.778 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled alertmanager update... 2024-06-01T02:37:38.126 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:37 smithi146 ceph-mon[25945]: Deploying daemon prometheus.a on smithi146 2024-06-01T02:37:38.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:37 smithi146 ceph-mon[25945]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi062=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:38.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:37 smithi146 ceph-mon[25945]: Saving service alertmanager spec with placement smithi062=a;count:1 2024-06-01T02:37:38.127 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[25095]: Deploying daemon prometheus.a on smithi146 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[25095]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi062=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[25095]: Saving service alertmanager spec with placement smithi062=a;count:1 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[29843]: Deploying daemon prometheus.a on smithi146 2024-06-01T02:37:38.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[29843]: from='client.24374 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi062=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:38.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[29843]: Saving service alertmanager spec with placement smithi062=a;count:1 2024-06-01T02:37:38.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:38.406 DEBUG:teuthology.orchestra.run.smithi062:alertmanager.a> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@alertmanager.a.service 2024-06-01T02:37:38.408 INFO:tasks.cephadm:Adding grafana.a on smithi146 2024-06-01T02:37:38.408 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph orch apply grafana '1;smithi146=a' 2024-06-01T02:37:40.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:39 smithi146 ceph-mon[25945]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:40.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:39 smithi062 ceph-mon[25095]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:40.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:39 smithi062 ceph-mon[29843]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:41.147 INFO:teuthology.orchestra.run.smithi146.stdout:Scheduled grafana update... 2024-06-01T02:37:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:41 smithi146 ceph-mon[25945]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:42.231 DEBUG:teuthology.orchestra.run.smithi146:grafana.a> sudo journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@grafana.a.service 2024-06-01T02:37:42.234 INFO:tasks.cephadm:Setting up client nodes... 2024-06-01T02:37:42.234 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-01T02:37:42.261 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:41 smithi062 ceph-mon[25095]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:42.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:42.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:41 smithi062 ceph-mon[29843]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:42.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:42.883 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:42 smithi062 ceph-mon[25095]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:42.883 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:42 smithi062 ceph-mon[25095]: Saving service grafana spec with placement smithi146=a;count:1 2024-06-01T02:37:42.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:42 smithi062 ceph-mon[29843]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:42.883 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:42 smithi062 ceph-mon[29843]: Saving service grafana spec with placement smithi146=a;count:1 2024-06-01T02:37:43.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:42 smithi146 ceph-mon[25945]: from='client.24380 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi146=a", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:37:43.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:42 smithi146 ceph-mon[25945]: Saving service grafana spec with placement smithi146=a;count:1 2024-06-01T02:37:43.857 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:43 smithi062 ceph-mon[25095]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:43.857 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:43 smithi062 ceph-mon[29843]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:44.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:43 smithi146 ceph-mon[25945]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:44.421 INFO:teuthology.orchestra.run.smithi062.stdout:[client.0] 2024-06-01T02:37:44.421 INFO:teuthology.orchestra.run.smithi062.stdout: key = AQB4iVpmx+EKGRAA6vCsVN3sqzhAHosxXwnOzw== 2024-06-01T02:37:44.994 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:44 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1140941292' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:44.994 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:44 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:44.994 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:44 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T02:37:44.994 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:44 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1140941292' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:44.995 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:44 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:44.995 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:44 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T02:37:45.019 DEBUG:teuthology.orchestra.run.smithi062:> set -ex 2024-06-01T02:37:45.019 DEBUG:teuthology.orchestra.run.smithi062:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-01T02:37:45.019 DEBUG:teuthology.orchestra.run.smithi062:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-01T02:37:45.100 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-01T02:37:45.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:44 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1140941292' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:45.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:44 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:45.122 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:44 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T02:37:46.105 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:45 smithi146 ceph-mon[25945]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:46.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:45 smithi062 ceph-mon[25095]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:46.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:45 smithi062 ceph-mon[29843]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:46.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:46 smithi146 ceph-mon[25945]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:47.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:46 smithi062 ceph-mon[25095]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:47.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:46 smithi062 ceph-mon[29843]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:47.986 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-06-01T02:37:47.986 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQB7iVpmjHyYOhAALhS2y1SN0fWS/cneyJoHDg== 2024-06-01T02:37:49.195 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:48 smithi146 ceph-mon[25945]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:49.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:48 smithi146 ceph-mon[25945]: from='client.? 172.21.15.146:0/1685533477' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:49.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:48 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:49.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:48 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T02:37:49.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[25095]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:49.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[25095]: from='client.? 172.21.15.146:0/1685533477' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:49.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:49.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T02:37:49.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[29843]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:49.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[29843]: from='client.? 172.21.15.146:0/1685533477' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:49.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-01T02:37:49.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:48 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-01T02:37:49.825 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-06-01T02:37:49.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-01T02:37:49.825 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-01T02:37:49.910 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-01T02:37:49.910 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-01T02:37:49.911 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-01T02:37:50.141 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:37:50.454 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 systemd[1]: Starting Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:37:50.837 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 podman[40977]: 2024-06-01 02:37:50.471175509 +0000 UTC m=+0.017935320 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-01T02:37:50.837 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 podman[40977]: 2024-06-01 02:37:50.578244931 +0000 UTC m=+0.125004736 container create 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:37:50.837 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 podman[40977]: 2024-06-01 02:37:50.838749758 +0000 UTC m=+0.385509562 container init 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:37:51.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-mon[25945]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:51.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 podman[40977]: 2024-06-01 02:37:50.841572827 +0000 UTC m=+0.388332644 container start 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:37:51.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.864Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-01T02:37:51.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.864Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-01T02:37:51.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.864Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.864Z caller=main.go:518 level=info host_details="(Linux 5.14.0-452.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024 x86_64 smithi146 (none))" 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.864Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.864Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.865Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.866Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.867Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.868Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.868Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.378µs 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.868Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.868Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.868Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.75µs wal_replay_duration=109.572µs total_replay_duration=141.519µs 2024-06-01T02:37:51.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.869Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-01T02:37:51.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.869Z caller=main.go:947 level=info msg="TSDB started" 2024-06-01T02:37:51.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.869Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-01T02:37:51.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.885Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.314683ms db_storage=637ns remote_storage=1.617µs web_handler=205ns query_engine=539ns scrape=159.417µs scrape_sd=13.638µs notify=696ns notify_sd=1.34µs rules=16.001876ms 2024-06-01T02:37:51.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:37:50.885Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-01T02:37:51.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 bash[40977]: 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 2024-06-01T02:37:51.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:37:50 smithi146 systemd[1]: Started Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:37:51.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:50 smithi062 ceph-mon[25095]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:51.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:50 smithi062 ceph-mon[29843]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:52.035 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:37:52.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:51 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:52.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:51 smithi062 ceph-mon[25095]: Deploying daemon node-exporter.a on smithi062 2024-06-01T02:37:52.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:51 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:52.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:51 smithi062 ceph-mon[29843]: Deploying daemon node-exporter.a on smithi062 2024-06-01T02:37:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:51 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:51 smithi146 ceph-mon[25945]: Deploying daemon node-exporter.a on smithi062 2024-06-01T02:37:52.708 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":20,"active_gid":24331,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6800","nonce":191358327},{"type":"v1","addr":"172.21.15.62:6801","nonce":191358327}]},"active_addr":"172.21.15.62:6801/191358327","active_change":"2024-06-01T02:37:33.124452+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24332,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.62:8443/","prometheus":"http://172.21.15.62:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":1350431142}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":872011279}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":51456829}]},{"addrvec":[{"type":"v2","addr":"172.21.15.62:0","nonce":313361820}]}]}} 2024-06-01T02:37:52.713 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-01T02:37:52.714 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-01T02:37:52.714 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-01T02:37:52.948 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:37:52.978 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:52 smithi062 bash[44504]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-06-01T02:37:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:52 smithi062 ceph-mon[25095]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:53.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:52 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3440556124' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-01T02:37:53.262 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:53 smithi062 bash[44504]: Getting image source signatures 2024-06-01T02:37:53.263 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:53 smithi062 bash[44504]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-06-01T02:37:53.263 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:53 smithi062 bash[44504]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-06-01T02:37:53.263 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:53 smithi062 bash[44504]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-06-01T02:37:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:52 smithi062 ceph-mon[29843]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:52 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3440556124' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-01T02:37:53.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:52 smithi146 ceph-mon[25945]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:53.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:52 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3440556124' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-01T02:37:55.096 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:54 smithi062 ceph-mon[29843]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:55.097 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:54 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:55.097 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:54 smithi062 bash[44504]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-06-01T02:37:55.097 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:54 smithi062 ceph-mon[25095]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:55.097 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:54 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:55.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:54 smithi146 ceph-mon[25945]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:55.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:54 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:55.432 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 bash[44504]: Writing manifest to image destination 2024-06-01T02:37:55.754 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 podman[44504]: 2024-06-01 02:37:55.40815174 +0000 UTC m=+3.328639593 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-01T02:37:55.754 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 podman[44504]: 2024-06-01 02:37:55.641361761 +0000 UTC m=+3.561849612 container create b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:37:56.129 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:37:56.129 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":60,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","created":"2024-06-01T02:32:14.312516+0000","modified":"2024-06-01T02:37:33.124098+0000","last_up_change":"2024-06-01T02:37:16.834197+0000","last_in_change":"2024-06-01T02:36:59.938186+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-01T02:35:20.229606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0e376b0d-882e-48c6-b47f-8d68368ff47b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6803","nonce":2961277760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6805","nonce":2961277760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6809","nonce":2961277760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6807","nonce":2961277760}]},"public_addr":"172.21.15.62:6803/2961277760","cluster_addr":"172.21.15.62:6805/2961277760","heartbeat_back_addr":"172.21.15.62:6809/2961277760","heartbeat_front_addr":"172.21.15.62:6807/2961277760","state":["exists","up"]},{"osd":1,"uuid":"3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6811","nonce":42392874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6813","nonce":42392874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6817","nonce":42392874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6815","nonce":42392874}]},"public_addr":"172.21.15.62:6811/42392874","cluster_addr":"172.21.15.62:6813/42392874","heartbeat_back_addr":"172.21.15.62:6817/42392874","heartbeat_front_addr":"172.21.15.62:6815/42392874","state":["exists","up"]},{"osd":2,"uuid":"c16629b6-e552-407a-b74b-d2beb921a2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6819","nonce":3316357651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6821","nonce":3316357651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6825","nonce":3316357651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6823","nonce":3316357651}]},"public_addr":"172.21.15.62:6819/3316357651","cluster_addr":"172.21.15.62:6821/3316357651","heartbeat_back_addr":"172.21.15.62:6825/3316357651","heartbeat_front_addr":"172.21.15.62:6823/3316357651","state":["exists","up"]},{"osd":3,"uuid":"71b115be-d78f-4df3-89c6-15fb5639c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6827","nonce":2184160246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6829","nonce":2184160246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6833","nonce":2184160246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6831","nonce":2184160246}]},"public_addr":"172.21.15.62:6827/2184160246","cluster_addr":"172.21.15.62:6829/2184160246","heartbeat_back_addr":"172.21.15.62:6833/2184160246","heartbeat_front_addr":"172.21.15.62:6831/2184160246","state":["exists","up"]},{"osd":4,"uuid":"6009746c-ef5e-4635-9990-75af41fe143b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6801","nonce":1882199247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6803","nonce":1882199247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6807","nonce":1882199247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6805","nonce":1882199247}]},"public_addr":"172.21.15.146:6801/1882199247","cluster_addr":"172.21.15.146:6803/1882199247","heartbeat_back_addr":"172.21.15.146:6807/1882199247","heartbeat_front_addr":"172.21.15.146:6805/1882199247","state":["exists","up"]},{"osd":5,"uuid":"7b07195b-3822-4c75-829f-4e43ac03536b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6809","nonce":2004940970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6811","nonce":2004940970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6815","nonce":2004940970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6813","nonce":2004940970}]},"public_addr":"172.21.15.146:6809/2004940970","cluster_addr":"172.21.15.146:6811/2004940970","heartbeat_back_addr":"172.21.15.146:6815/2004940970","heartbeat_front_addr":"172.21.15.146:6813/2004940970","state":["exists","up"]},{"osd":6,"uuid":"c3fff939-031b-4431-9518-b90185765c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6817","nonce":3392045728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6819","nonce":3392045728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6823","nonce":3392045728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6821","nonce":3392045728}]},"public_addr":"172.21.15.146:6817/3392045728","cluster_addr":"172.21.15.146:6819/3392045728","heartbeat_back_addr":"172.21.15.146:6823/3392045728","heartbeat_front_addr":"172.21.15.146:6821/3392045728","state":["exists","up"]},{"osd":7,"uuid":"06ee51a3-2d2b-4636-b660-a74b6aeaafb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6825","nonce":2912039056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6827","nonce":2912039056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6831","nonce":2912039056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6829","nonce":2912039056}]},"public_addr":"172.21.15.146:6825/2912039056","cluster_addr":"172.21.15.146:6827/2912039056","heartbeat_back_addr":"172.21.15.146:6831/2912039056","heartbeat_front_addr":"172.21.15.146:6829/2912039056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:34:28.509822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:34:52.874848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:35:16.346228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:35:42.435488+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:04.277597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:27.522336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:51.282685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:37:15.589252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6801/779526503":"2024-06-02T02:37:33.124049+0000","172.21.15.62:0/1986861472":"2024-06-02T02:37:33.124049+0000","172.21.15.62:6800/779526503":"2024-06-02T02:37:33.124049+0000","172.21.15.62:0/1694477920":"2024-06-02T02:37:33.124049+0000","172.21.15.62:0/1920187738":"2024-06-02T02:37:33.124049+0000","172.21.15.62:6801/3339555501":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/4106139753":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3491871":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/1842014486":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3141512320":"2024-06-02T02:37:33.124049+0000","172.21.15.62:6800/1434776916":"2024-06-02T02:32:46.207475+0000","172.21.15.62:0/1726234481":"2024-06-02T02:32:46.207475+0000","172.21.15.62:6801/1434776916":"2024-06-02T02:32:46.207475+0000","172.21.15.62:0/2403150514":"2024-06-02T02:32:46.207475+0000","172.21.15.62:6800/3339555501":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3889207289":"2024-06-02T02:32:46.207475+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-01T02:37:56.129 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 podman[44504]: 2024-06-01 02:37:55.943872933 +0000 UTC m=+3.864360792 container init b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 podman[44504]: 2024-06-01 02:37:55.947028738 +0000 UTC m=+3.867516598 container start b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.951Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.951Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.951Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.951Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=arp 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-01T02:37:56.130 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=edac 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-01T02:37:56.131 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=os 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-01T02:37:56.132 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=stat 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=time 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=uname 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-01T02:37:56.133 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:55 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[44675]: ts=2024-06-01T02:37:55.952Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-01T02:37:56.512 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:56 smithi062 bash[44504]: b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 2024-06-01T02:37:56.512 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:37:56 smithi062 systemd[1]: Started Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:37:56.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:37:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:37:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:37:56.805 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-01T02:37:56.805 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-01T02:37:56.888 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:56 smithi146 systemd[1]: Starting Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:37:57.030 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:37:57.153 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[25095]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:57.153 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1662612678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:57.154 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:57.154 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[25095]: Deploying daemon node-exporter.b on smithi146 2024-06-01T02:37:57.154 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[29843]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:57.154 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1662612678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:57.154 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:57.154 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:56 smithi062 ceph-mon[29843]: Deploying daemon node-exporter.b on smithi146 2024-06-01T02:37:57.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:56 smithi146 ceph-mon[25945]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:57.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:56 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1662612678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:57.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:56 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:57.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:56 smithi146 ceph-mon[25945]: Deploying daemon node-exporter.b on smithi146 2024-06-01T02:37:57.205 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:57 smithi146 bash[41210]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-06-01T02:37:57.705 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:57 smithi146 bash[41210]: Getting image source signatures 2024-06-01T02:37:57.705 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:57 smithi146 bash[41210]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-06-01T02:37:57.705 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:57 smithi146 bash[41210]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-06-01T02:37:57.705 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:57 smithi146 bash[41210]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-06-01T02:37:58.955 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:58 smithi146 bash[41210]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-06-01T02:37:58.955 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:58 smithi146 bash[41210]: Writing manifest to image destination 2024-06-01T02:37:59.052 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:37:59.053 INFO:teuthology.orchestra.run.smithi062.stdout:{"epoch":60,"fsid":"176c65a6-1fbf-11ef-bc9b-c7b262605968","created":"2024-06-01T02:32:14.312516+0000","modified":"2024-06-01T02:37:33.124098+0000","last_up_change":"2024-06-01T02:37:16.834197+0000","last_in_change":"2024-06-01T02:36:59.938186+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-01T02:35:20.229606+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"0e376b0d-882e-48c6-b47f-8d68368ff47b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6802","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6803","nonce":2961277760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6804","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6805","nonce":2961277760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6808","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6809","nonce":2961277760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6806","nonce":2961277760},{"type":"v1","addr":"172.21.15.62:6807","nonce":2961277760}]},"public_addr":"172.21.15.62:6803/2961277760","cluster_addr":"172.21.15.62:6805/2961277760","heartbeat_back_addr":"172.21.15.62:6809/2961277760","heartbeat_front_addr":"172.21.15.62:6807/2961277760","state":["exists","up"]},{"osd":1,"uuid":"3ad3ee9a-b899-40b3-bbb6-e84b0cb887b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6810","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6811","nonce":42392874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6812","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6813","nonce":42392874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6816","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6817","nonce":42392874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6814","nonce":42392874},{"type":"v1","addr":"172.21.15.62:6815","nonce":42392874}]},"public_addr":"172.21.15.62:6811/42392874","cluster_addr":"172.21.15.62:6813/42392874","heartbeat_back_addr":"172.21.15.62:6817/42392874","heartbeat_front_addr":"172.21.15.62:6815/42392874","state":["exists","up"]},{"osd":2,"uuid":"c16629b6-e552-407a-b74b-d2beb921a2df","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6818","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6819","nonce":3316357651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6820","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6821","nonce":3316357651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6824","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6825","nonce":3316357651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6822","nonce":3316357651},{"type":"v1","addr":"172.21.15.62:6823","nonce":3316357651}]},"public_addr":"172.21.15.62:6819/3316357651","cluster_addr":"172.21.15.62:6821/3316357651","heartbeat_back_addr":"172.21.15.62:6825/3316357651","heartbeat_front_addr":"172.21.15.62:6823/3316357651","state":["exists","up"]},{"osd":3,"uuid":"71b115be-d78f-4df3-89c6-15fb5639c913","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6826","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6827","nonce":2184160246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6828","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6829","nonce":2184160246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6832","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6833","nonce":2184160246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.62:6830","nonce":2184160246},{"type":"v1","addr":"172.21.15.62:6831","nonce":2184160246}]},"public_addr":"172.21.15.62:6827/2184160246","cluster_addr":"172.21.15.62:6829/2184160246","heartbeat_back_addr":"172.21.15.62:6833/2184160246","heartbeat_front_addr":"172.21.15.62:6831/2184160246","state":["exists","up"]},{"osd":4,"uuid":"6009746c-ef5e-4635-9990-75af41fe143b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6801","nonce":1882199247}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6803","nonce":1882199247}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6807","nonce":1882199247}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":1882199247},{"type":"v1","addr":"172.21.15.146:6805","nonce":1882199247}]},"public_addr":"172.21.15.146:6801/1882199247","cluster_addr":"172.21.15.146:6803/1882199247","heartbeat_back_addr":"172.21.15.146:6807/1882199247","heartbeat_front_addr":"172.21.15.146:6805/1882199247","state":["exists","up"]},{"osd":5,"uuid":"7b07195b-3822-4c75-829f-4e43ac03536b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6809","nonce":2004940970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6810","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6811","nonce":2004940970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6815","nonce":2004940970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":2004940970},{"type":"v1","addr":"172.21.15.146:6813","nonce":2004940970}]},"public_addr":"172.21.15.146:6809/2004940970","cluster_addr":"172.21.15.146:6811/2004940970","heartbeat_back_addr":"172.21.15.146:6815/2004940970","heartbeat_front_addr":"172.21.15.146:6813/2004940970","state":["exists","up"]},{"osd":6,"uuid":"c3fff939-031b-4431-9518-b90185765c3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6816","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6817","nonce":3392045728}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6819","nonce":3392045728}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6823","nonce":3392045728}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6820","nonce":3392045728},{"type":"v1","addr":"172.21.15.146:6821","nonce":3392045728}]},"public_addr":"172.21.15.146:6817/3392045728","cluster_addr":"172.21.15.146:6819/3392045728","heartbeat_back_addr":"172.21.15.146:6823/3392045728","heartbeat_front_addr":"172.21.15.146:6821/3392045728","state":["exists","up"]},{"osd":7,"uuid":"06ee51a3-2d2b-4636-b660-a74b6aeaafb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6825","nonce":2912039056}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6827","nonce":2912039056}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6831","nonce":2912039056}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2912039056},{"type":"v1","addr":"172.21.15.146:6829","nonce":2912039056}]},"public_addr":"172.21.15.146:6825/2912039056","cluster_addr":"172.21.15.146:6827/2912039056","heartbeat_back_addr":"172.21.15.146:6831/2912039056","heartbeat_front_addr":"172.21.15.146:6829/2912039056","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:34:28.509822+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:34:52.874848+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:35:16.346228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:35:42.435488+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:04.277597+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:27.522336+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:36:51.282685+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-06-01T02:37:15.589252+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.62:6801/779526503":"2024-06-02T02:37:33.124049+0000","172.21.15.62:0/1986861472":"2024-06-02T02:37:33.124049+0000","172.21.15.62:6800/779526503":"2024-06-02T02:37:33.124049+0000","172.21.15.62:0/1694477920":"2024-06-02T02:37:33.124049+0000","172.21.15.62:0/1920187738":"2024-06-02T02:37:33.124049+0000","172.21.15.62:6801/3339555501":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/4106139753":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3491871":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/1842014486":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3141512320":"2024-06-02T02:37:33.124049+0000","172.21.15.62:6800/1434776916":"2024-06-02T02:32:46.207475+0000","172.21.15.62:0/1726234481":"2024-06-02T02:32:46.207475+0000","172.21.15.62:6801/1434776916":"2024-06-02T02:32:46.207475+0000","172.21.15.62:0/2403150514":"2024-06-02T02:32:46.207475+0000","172.21.15.62:6800/3339555501":"2024-06-02T02:33:09.305931+0000","172.21.15.62:0/3889207289":"2024-06-02T02:32:46.207475+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 podman[41210]: 2024-06-01 02:37:59.116907711 +0000 UTC m=+2.125387652 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 podman[41210]: 2024-06-01 02:37:59.215644242 +0000 UTC m=+2.224124175 container create bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 podman[41210]: 2024-06-01 02:37:59.434468337 +0000 UTC m=+2.442948280 container init bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 podman[41210]: 2024-06-01 02:37:59.43748384 +0000 UTC m=+2.445963782 container start bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.441Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.441Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-01T02:37:59.542 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=arp 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-01T02:37:59.543 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.442Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=edac 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-01T02:37:59.544 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=os 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-01T02:37:59.545 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=stat 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=time 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=uname 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-01T02:37:59.546 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[41261]: ts=2024-06-01T02:37:59.443Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-01T02:37:59.547 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 bash[41210]: bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 2024-06-01T02:37:59.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-01T02:37:59.730 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-01T02:37:59.731 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-01T02:37:59.731 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-01T02:37:59.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-01T02:37:59.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-01T02:37:59.732 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-06-01T02:37:59.733 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-06-01T02:37:59.955 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:37:59 smithi146 systemd[1]: Started Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:37:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-mon[25945]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:37:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1985351559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:37:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:37:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:37:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:37:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:37:59 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:38:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[25095]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1985351559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:38:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:38:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:38:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:38:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[29843]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1985351559' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-01T02:38:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:38:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:38:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:37:59 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:38:00.144 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:00.145 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:00.205 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:00.213 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:00.215 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:00.219 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:01.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:38:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:00] "GET /metrics HTTP/1.1" 200 191530 "" "Prometheus/2.33.4" 2024-06-01T02:38:02.096 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:01 smithi062 ceph-mon[25095]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:02.097 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:01 smithi062 ceph-mon[29843]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:02.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:01 smithi146 ceph-mon[25945]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:03.079 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:03.235 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:03 smithi062 ceph-mon[25095]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:03.236 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:03 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:03.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:03 smithi062 ceph-mon[29843]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:03.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:03 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:03.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:03 smithi146 ceph-mon[25945]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:03.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:03 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:05.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:04 smithi062 ceph-mon[25095]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:05.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:04 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:05.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:04 smithi062 ceph-mon[29843]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:05.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:04 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:04 smithi146 ceph-mon[25945]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:04 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:06.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:38:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:38:07.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:06 smithi062 ceph-mon[25095]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:07.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:06 smithi062 ceph-mon[29843]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:07.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:06 smithi146 ceph-mon[25945]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:07.253 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:08.612 INFO:teuthology.orchestra.run.smithi062.stdout:34359738414 2024-06-01T02:38:08.612 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-01T02:38:09.410 INFO:teuthology.orchestra.run.smithi062.stdout:55834574889 2024-06-01T02:38:09.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-01T02:38:09.956 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:09 smithi062 ceph-mon[25095]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:09.957 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:09 smithi062 ceph-mon[29843]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:10.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:09 smithi146 ceph-mon[25945]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:10.568 INFO:teuthology.orchestra.run.smithi062.stdout:167503724565 2024-06-01T02:38:10.569 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-01T02:38:11.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:38:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:10] "GET /metrics HTTP/1.1" 200 191508 "" "Prometheus/2.33.4" 2024-06-01T02:38:11.358 INFO:teuthology.orchestra.run.smithi062.stdout:111669149727 2024-06-01T02:38:11.359 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-01T02:38:11.973 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:11 smithi062 ceph-mon[25095]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:11.974 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:11 smithi062 ceph-mon[29843]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:12.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:11 smithi146 ceph-mon[25945]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:13.824 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:13.825 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:13.825 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:13.825 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:14.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:13 smithi146 ceph-mon[25945]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:14.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:13 smithi062 ceph-mon[25095]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:14.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:13 smithi062 ceph-mon[29843]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:14.329 INFO:teuthology.orchestra.run.smithi062.stdout:133143986203 2024-06-01T02:38:14.330 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-01T02:38:14.787 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:15.563 INFO:teuthology.orchestra.run.smithi062.stdout:77309411364 2024-06-01T02:38:15.564 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-01T02:38:16.196 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:15 smithi062 ceph-mon[25095]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:16.196 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:15 smithi062 ceph-mon[29843]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:16.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:15 smithi146 ceph-mon[25945]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:16.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:38:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:38:17.947 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:17 smithi062 ceph-mon[25095]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:17.947 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:17 smithi062 ceph-mon[29843]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:17 smithi146 ceph-mon[25945]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:20.107 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:19 smithi062 ceph-mon[25095]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:20.107 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:19 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:20.107 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:19 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:20.108 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:19 smithi062 ceph-mon[29843]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:20.108 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:19 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:20.108 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:19 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:19 smithi146 ceph-mon[25945]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:19 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:19 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:20.744 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:38:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:20] "GET /metrics HTTP/1.1" 200 191501 "" "Prometheus/2.33.4" 2024-06-01T02:38:20.768 INFO:teuthology.orchestra.run.smithi062.stdout:236223201292 2024-06-01T02:38:20.768 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-06-01T02:38:20.996 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:20 smithi062 ceph-mon[25095]: Deploying daemon alertmanager.a on smithi062 2024-06-01T02:38:21.042 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:20 smithi062 ceph-mon[29843]: Deploying daemon alertmanager.a on smithi062 2024-06-01T02:38:21.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:20 smithi146 ceph-mon[25945]: Deploying daemon alertmanager.a on smithi062 2024-06-01T02:38:21.382 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:21.990 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:21 smithi062 ceph-mon[25095]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:21.991 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:21 smithi062 ceph-mon[29843]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:22.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:21 smithi146 ceph-mon[25945]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:23.039 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:22 smithi062 ceph-mon[25095]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:23.039 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:22 smithi062 ceph-mon[29843]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:23.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:22 smithi146 ceph-mon[25945]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:23.943 INFO:teuthology.orchestra.run.smithi062.stdout:55834574892 2024-06-01T02:38:23.949 INFO:teuthology.orchestra.run.smithi062.stdout:111669149730 2024-06-01T02:38:25.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:24 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4220537668' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-01T02:38:25.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:24 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/641412884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-01T02:38:25.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:24 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4220537668' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-01T02:38:25.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:24 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/641412884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-01T02:38:25.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:24 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4220537668' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-01T02:38:25.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:24 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/641412884' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-01T02:38:25.449 INFO:teuthology.orchestra.run.smithi062.stdout:34359738417 2024-06-01T02:38:25.872 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574892 for osd.1 2024-06-01T02:38:25.872 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:26.057 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:25 smithi062 ceph-mon[25095]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:26.058 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:25 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/615914866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-01T02:38:26.058 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:25 smithi062 ceph-mon[29843]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:26.058 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:25 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/615914866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-01T02:38:26.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:25 smithi146 ceph-mon[25945]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:26.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:25 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/615914866' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-01T02:38:26.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:38:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:38:26.807 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149730 for osd.3 2024-06-01T02:38:26.807 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:28.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:27 smithi062 ceph-mon[25095]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:28.113 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:27 smithi062 ceph-mon[29843]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:27 smithi146 ceph-mon[25945]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:29.097 INFO:teuthology.orchestra.run.smithi062.stdout:77309411367 2024-06-01T02:38:29.535 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738417 for osd.0 2024-06-01T02:38:29.536 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:30.099 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:29 smithi062 ceph-mon[25095]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:30.100 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:29 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4173018108' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-01T02:38:30.100 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:29 smithi062 ceph-mon[29843]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:30.100 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:29 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4173018108' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-01T02:38:30.105 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:30.128 INFO:teuthology.orchestra.run.smithi062.stdout:201863462931 2024-06-01T02:38:30.128 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-06-01T02:38:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:29 smithi146 ceph-mon[25945]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:29 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4173018108' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-01T02:38:31.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:38:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:30] "GET /metrics HTTP/1.1" 200 191501 "" "Prometheus/2.33.4" 2024-06-01T02:38:31.746 INFO:teuthology.orchestra.run.smithi062.stdout:236223201294 2024-06-01T02:38:32.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:31 smithi062 ceph-mon[29843]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:32.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:31 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3225091535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-01T02:38:32.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:31 smithi062 ceph-mon[25095]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:32.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:31 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3225091535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-01T02:38:32.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:31 smithi146 ceph-mon[25945]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:32.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:31 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3225091535' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-06-01T02:38:32.732 INFO:teuthology.orchestra.run.smithi062.stdout:133143986207 2024-06-01T02:38:32.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411367 for osd.2 2024-06-01T02:38:32.892 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:32 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4794241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-01T02:38:33.253 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:32 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4794241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-01T02:38:33.253 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:32 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4794241' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-01T02:38:33.412 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986207 for osd.4 2024-06-01T02:38:33.413 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:33.958 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:33 smithi062 ceph-mon[25095]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:33.959 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:33 smithi062 ceph-mon[29843]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:33.982 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201292 got 236223201294 for osd.7 2024-06-01T02:38:33.982 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:34.115 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:34.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:33 smithi146 ceph-mon[25945]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:34.961 INFO:teuthology.orchestra.run.smithi062.stdout:167503724570 2024-06-01T02:38:35.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:34 smithi146 ceph-mon[25945]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:35.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:38:35.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:38:35.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:38:35.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:38:35.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[29843]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:35.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:38:35.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[25095]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:38:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:38:36.148 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:35 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3921455213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-01T02:38:36.148 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:35 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3921455213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-01T02:38:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:35 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3921455213' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-01T02:38:36.647 INFO:teuthology.orchestra.run.smithi062.stdout:201863462933 2024-06-01T02:38:36.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:38:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:38:37.147 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:36 smithi062 ceph-mon[25095]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:37.147 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:36 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2056864540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-01T02:38:37.148 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:36 smithi062 ceph-mon[29843]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:37.148 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:36 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2056864540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-01T02:38:37.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:36 smithi146 ceph-mon[25945]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:37.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:36 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2056864540' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-06-01T02:38:37.425 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462931 got 201863462933 for osd.6 2024-06-01T02:38:37.426 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:37.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724565 got 167503724570 for osd.5 2024-06-01T02:38:37.998 DEBUG:teuthology.parallel:result is None 2024-06-01T02:38:37.998 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-01T02:38:37.998 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-01T02:38:38.778 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:40.063 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:39 smithi062 ceph-mon[25095]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:40.063 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:39 smithi062 ceph-mon[29843]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:40.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:39 smithi146 ceph-mon[25945]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:40.610 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:38:40.611 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-06-01T02:38:41.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:38:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:40] "GET /metrics HTTP/1.1" 200 191502 "" "Prometheus/2.33.4" 2024-06-01T02:38:41.408 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-06-01T02:38:39.882455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48892,"kb_used_data":6140,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912324,"statfs":{"total":767960285184,"available":767910219776,"internally_reserved":0,"allocated":6287360,"data_stored":3510093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002321"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-06-01T02:37:33.960194+0000","last_change":"2024-06-01T02:37:21.093612+0000","last_active":"2024-06-01T02:37:33.960194+0000","last_peered":"2024-06-01T02:37:33.960194+0000","last_clean":"2024-06-01T02:37:33.960194+0000","last_became_active":"2024-06-01T02:37:21.093195+0000","last_became_peered":"2024-06-01T02:37:21.093195+0000","last_unstale":"2024-06-01T02:37:33.960194+0000","last_undegraded":"2024-06-01T02:37:33.960194+0000","last_fullsized":"2024-06-01T02:37:33.960194+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-01T02:35:20.279796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-01T02:35:20.279796+0000","last_clean_scrub_stamp":"2024-06-01T02:35:20.279796+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-02T06:12:33.656664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201296,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.375}]},{"osd":1,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.432},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.45900000000000002}]},{"osd":2,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.503}]},{"osd":3,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.53000000000000003}]},{"osd":4,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.54900000000000004}]},{"osd":5,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.54100000000000004}]},{"osd":6,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60199999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.68000000000000005}]},{"osd":1,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.624}]},{"osd":4,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Jun 1 02:38:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.375,"15min":0.35499999999999998},"min":{"1min":0.248,"5min":0.214,"15min":0.214},"max":{"1min":0.69499999999999995,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.40300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63400000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Jun 1 02:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.57899999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Jun 1 02:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.437,"15min":0.433},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.44500000000000001},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.39300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Jun 1 02:38:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61799999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Jun 1 02:37:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Jun 1 02:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.61199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 1 02:37:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.38600000000000001,"15min":0.36899999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Jun 1 02:38:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.623},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.40300000000000002},"min":{"1min":0.27500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60999999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Jun 1 02:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.70399999999999996}]},{"osd":4,"last update":"Sat Jun 1 02:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65000000000000002,"15min":0.64800000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91900000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Jun 1 02:38:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.877,"15min":0.877},"last":1.014},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79300000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":1.115}]},{"osd":6,"last update":"Sat Jun 1 02:37:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":1.131},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":1.1379999999999999}]},{"osd":7,"last update":"Sat Jun 1 02:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":1.165},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":1.038}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6212,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41399999999999998,"15min":0.40799999999999997},"min":{"1min":0.26200000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.495},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.42499999999999999},"min":{"1min":0.27800000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36799999999999999}]},{"osd":1,"last update":"Sat Jun 1 02:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63700000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46400000000000002,"15min":0.45800000000000002},"min":{"1min":0.315,"5min":0.251,"15min":0.251},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jun 1 02:37:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71499999999999997,"5min":0.748,"15min":0.748},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Jun 1 02:38:15 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.59399999999999997,"15min":0.58799999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88500000000000001,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.622,"15min":0.61399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Jun 1 02:38:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58599999999999997,"15min":0.57399999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.46100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.749}]},{"osd":6,"last update":"Sat Jun 1 02:37:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Jun 1 02:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68000000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.42399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51900000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58299999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jun 1 02:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.316,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.54800000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Jun 1 02:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.48699999999999999,"15min":0.49399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56399999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.46600000000000003,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.54500000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Jun 1 02:38:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.81799999999999995}]},{"osd":5,"last update":"Sat Jun 1 02:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.626},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59599999999999997,"15min":0.58599999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78000000000000003}]},{"osd":6,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.62},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Jun 1 02:38:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.745}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57199999999999995,"15min":0.58099999999999996},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70399999999999996,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.40000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.625},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71899999999999997,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57699999999999996,"15min":0.58199999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Jun 1 02:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.33100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64000000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38100000000000001}]},{"osd":6,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.435},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.373}]},{"osd":7,"last update":"Sat Jun 1 02:38:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.40100000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":239807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52000000000000002,"15min":0.52400000000000002},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63700000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.625},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67800000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.41499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42599999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.72699999999999998}]},{"osd":2,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.71999999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.42199999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.64700000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69299999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.82799999999999996}]},{"osd":4,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.78900000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.44400000000000001,"15min":0.434},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Jun 1 02:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-01T02:38:41.411 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-01T02:38:41.656 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:42.039 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:42.039 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:38:42.039 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.039 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.039 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:38:42.040 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: from='client.24496 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:38:42.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:42.934 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:42 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:38:42.935 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:42 smithi062 ceph-mon[25095]: Deploying daemon grafana.a on smithi146 2024-06-01T02:38:42.935 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:42 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:38:42.935 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:42 smithi062 ceph-mon[29843]: Deploying daemon grafana.a on smithi146 2024-06-01T02:38:43.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:42 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:38:43.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:42 smithi146 ceph-mon[25945]: Deploying daemon grafana.a on smithi146 2024-06-01T02:38:43.327 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:38:43.329 INFO:teuthology.orchestra.run.smithi062.stderr:dumped all 2024-06-01T02:38:43.670 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:38:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[47864]: level=info ts=2024-06-01T02:38:43.480Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000931904s 2024-06-01T02:38:43.987 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:43 smithi062 ceph-mon[25095]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:43.987 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:43 smithi062 ceph-mon[29843]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:44.012 INFO:teuthology.orchestra.run.smithi062.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-06-01T02:38:41.882771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48892,"kb_used_data":6140,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":749912324,"statfs":{"total":767960285184,"available":767910219776,"internally_reserved":0,"allocated":6287360,"data_stored":3510093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002284"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-06-01T02:37:33.960194+0000","last_change":"2024-06-01T02:37:21.093612+0000","last_active":"2024-06-01T02:37:33.960194+0000","last_peered":"2024-06-01T02:37:33.960194+0000","last_clean":"2024-06-01T02:37:33.960194+0000","last_became_active":"2024-06-01T02:37:21.093195+0000","last_became_peered":"2024-06-01T02:37:21.093195+0000","last_unstale":"2024-06-01T02:37:33.960194+0000","last_undegraded":"2024-06-01T02:37:33.960194+0000","last_fullsized":"2024-06-01T02:37:33.960194+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-01T02:35:20.279796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-01T02:35:20.279796+0000","last_clean_scrub_stamp":"2024-06-01T02:35:20.279796+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-02T06:12:33.656664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201297,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66000000000000003}]},{"osd":1,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.374}]},{"osd":5,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.53100000000000003}]},{"osd":6,"last update":"Sat Jun 1 02:38:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.622},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.67300000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.68000000000000005}]},{"osd":1,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.66000000000000003}]},{"osd":2,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.52600000000000002}]},{"osd":3,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.624}]},{"osd":4,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.59399999999999997}]},{"osd":5,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.54200000000000004}]},{"osd":7,"last update":"Sat Jun 1 02:38:17 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6340,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738812,"statfs":{"total":95995035648,"available":95988543488,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.375,"15min":0.35499999999999998},"min":{"1min":0.248,"5min":0.214,"15min":0.214},"max":{"1min":0.69499999999999995,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.40300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.63400000000000001,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.36299999999999999}]},{"osd":2,"last update":"Sat Jun 1 02:38:23 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.40300000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.40899999999999997,"15min":0.40300000000000002},"min":{"1min":0.28699999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.57899999999999996,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.40500000000000003}]},{"osd":3,"last update":"Sat Jun 1 02:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.437,"15min":0.433},"min":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.44800000000000001,"15min":0.44500000000000001},"min":{"1min":0.33500000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.29599999999999999}]},{"osd":4,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55200000000000005,"15min":0.54400000000000004},"min":{"1min":0.39300000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.58399999999999996,"15min":0.57299999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64600000000000002}]},{"osd":5,"last update":"Sat Jun 1 02:38:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.61799999999999999,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59399999999999997,"15min":0.59299999999999997},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.50600000000000001}]},{"osd":6,"last update":"Sat Jun 1 02:37:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51600000000000001}]},{"osd":7,"last update":"Sat Jun 1 02:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.61199999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 1 02:37:57 2024","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.374,"15min":0.35899999999999999},"min":{"1min":0.254,"5min":0.216,"15min":0.216},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.38600000000000001,"15min":0.36899999999999999},"min":{"1min":0.28999999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.57899999999999996}]},{"osd":2,"last update":"Sat Jun 1 02:38:24 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.318,"5min":0.249,"15min":0.249},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.623},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.41099999999999998,"15min":0.40300000000000002},"min":{"1min":0.27500000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.60999999999999999,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Jun 1 02:37:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45200000000000001,"15min":0.45100000000000001},"min":{"1min":0.34399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.70399999999999996}]},{"osd":4,"last update":"Sat Jun 1 02:38:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.65000000000000002,"15min":0.64800000000000002},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.91900000000000004,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Jun 1 02:38:31 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.627},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.82299999999999995,"5min":0.877,"15min":0.877},"last":1.014},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61499999999999999,"15min":0.61699999999999999},"min":{"1min":0.434,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79300000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":1.115}]},{"osd":6,"last update":"Sat Jun 1 02:37:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.865,"5min":1.865,"15min":1.865},"last":1.131},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":1.1379999999999999}]},{"osd":7,"last update":"Sat Jun 1 02:38:19 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":1.165},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":1.038}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6212,"kb_used_data":572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738940,"statfs":{"total":95995035648,"available":95988674560,"internally_reserved":0,"allocated":585728,"data_stored":240395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41399999999999998,"15min":0.40799999999999997},"min":{"1min":0.26200000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.495},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.42799999999999999,"15min":0.42499999999999999},"min":{"1min":0.27800000000000002,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.36799999999999999}]},{"osd":1,"last update":"Sat Jun 1 02:38:26 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63700000000000001,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46400000000000002,"15min":0.45800000000000002},"min":{"1min":0.315,"5min":0.251,"15min":0.251},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.46200000000000002}]},{"osd":3,"last update":"Sat Jun 1 02:37:53 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.495,"15min":0.496},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71499999999999997,"5min":0.748,"15min":0.748},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.51500000000000001,"15min":0.51700000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.52800000000000002}]},{"osd":4,"last update":"Sat Jun 1 02:38:15 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.59399999999999997,"15min":0.58799999999999997},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.88500000000000001,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.622,"15min":0.61399999999999999},"min":{"1min":0.48999999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.76300000000000001}]},{"osd":5,"last update":"Sat Jun 1 02:38:35 2024","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.58599999999999997,"15min":0.57399999999999995},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.64500000000000002,"15min":0.63900000000000001},"min":{"1min":0.46100000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.749}]},{"osd":6,"last update":"Sat Jun 1 02:37:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.71499999999999997}]},{"osd":7,"last update":"Sat Jun 1 02:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68000000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41899999999999998,"15min":0.42399999999999999},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.51900000000000002,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.58299999999999996,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.34999999999999998}]},{"osd":1,"last update":"Sat Jun 1 02:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.47899999999999998,"15min":0.48299999999999998},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44900000000000001,"15min":0.45300000000000001},"min":{"1min":0.316,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.54800000000000004,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.36599999999999999}]},{"osd":2,"last update":"Sat Jun 1 02:37:47 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.48699999999999999,"15min":0.49399999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56399999999999995,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.46600000000000003,"15min":0.47399999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.54500000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.29299999999999998}]},{"osd":4,"last update":"Sat Jun 1 02:38:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58799999999999997,"15min":0.58699999999999997},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"last":0.81799999999999995}]},{"osd":5,"last update":"Sat Jun 1 02:38:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.626},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59599999999999997,"15min":0.58599999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.78000000000000003}]},{"osd":6,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.62},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Jun 1 02:38:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.623,"5min":0.623,"15min":0.623},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.745}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57499999999999996,"15min":0.58599999999999997},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76900000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60799999999999998}]},{"osd":1,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.57199999999999995,"15min":0.58099999999999996},"min":{"1min":0.372,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70399999999999996,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52400000000000002,"15min":0.52500000000000002},"min":{"1min":0.40000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.57999999999999996}]},{"osd":2,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.625},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.52800000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.71899999999999997,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53700000000000003}]},{"osd":3,"last update":"Sat Jun 1 02:38:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54400000000000004,"15min":0.54600000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57699999999999996,"15min":0.58199999999999996},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.59499999999999997}]},{"osd":5,"last update":"Sat Jun 1 02:38:34 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.41399999999999998,"15min":0.40999999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.33100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.64000000000000001,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.38100000000000001}]},{"osd":6,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.435},{"interface":"front","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.373}]},{"osd":7,"last update":"Sat Jun 1 02:38:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.40100000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5692,"kb_used_data":564,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739460,"statfs":{"total":95995035648,"available":95989207040,"internally_reserved":0,"allocated":577536,"data_stored":239807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.52000000000000002,"15min":0.52400000000000002},"min":{"1min":0.39400000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.63700000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.625},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53900000000000003,"15min":0.54100000000000004},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67800000000000005,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.68799999999999994}]},{"osd":1,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.53400000000000003,"15min":0.52700000000000002},"min":{"1min":0.41499999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.73899999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.42599999999999999,"5min":0.374,"15min":0.374},"max":{"1min":0.71399999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.72699999999999998}]},{"osd":2,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.54100000000000004,"15min":0.54600000000000004},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.64700000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.40600000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.71999999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.79700000000000004}]},{"osd":3,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53700000000000003,"15min":0.53800000000000003},"min":{"1min":0.42199999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.64700000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.57699999999999996,"15min":0.57499999999999996},"min":{"1min":0.47799999999999998,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69299999999999995,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.82799999999999996}]},{"osd":4,"last update":"Sat Jun 1 02:38:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.436,"15min":0.42099999999999999},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.78900000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.44400000000000001,"15min":0.434},"min":{"1min":0.32000000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69799999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57399999999999995}]},{"osd":6,"last update":"Sat Jun 1 02:37:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.65600000000000003}]},{"osd":7,"last update":"Sat Jun 1 02:38:20 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-01T02:38:44.013 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-01T02:38:44.013 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-01T02:38:44.013 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-01T02:38:44.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-01T02:38:44.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:43 smithi146 ceph-mon[25945]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:44.234 INFO:teuthology.orchestra.run.smithi062.stderr:Inferring config /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/mon.a/config 2024-06-01T02:38:45.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:44 smithi062 ceph-mon[25095]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:38:45.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:45.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:44 smithi062 ceph-mon[29843]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:38:45.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:45.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:44 smithi146 ceph-mon[25945]: from='client.24502 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-01T02:38:45.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:46.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:45 smithi062 ceph-mon[25095]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:46.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:45 smithi062 ceph-mon[29843]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:46.060 INFO:teuthology.orchestra.run.smithi062.stdout: 2024-06-01T02:38:46.060 INFO:teuthology.orchestra.run.smithi062.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-01T02:38:46.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:45 smithi146 ceph-mon[25945]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:46.619 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-06-01T02:38:46.619 INFO:tasks.cephadm:Setup complete, yielding 2024-06-01T02:38:46.619 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-01T02:38:46.629 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-06-01T02:38:46.629 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-06-01T02:38:46.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:38:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:38:46.944 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:46 smithi062 ceph-mon[25095]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:46.944 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:46 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1770158107' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T02:38:46.944 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:46 smithi062 ceph-mon[29843]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:46.945 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:46 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1770158107' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T02:38:47.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:46 smithi146 ceph-mon[25945]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:47.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:46 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1770158107' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-01T02:38:48.981 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-01T02:38:48.994 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-06-01T02:38:48.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-06-01T02:38:49.552 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:49 smithi062 ceph-mon[25095]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:49.553 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:49 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' 2024-06-01T02:38:49.553 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:49 smithi062 ceph-mon[29843]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:49.553 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:49 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' 2024-06-01T02:38:49.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:49 smithi146 ceph-mon[25945]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:49.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:49 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' 2024-06-01T02:38:50.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:38:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:50] "GET /metrics HTTP/1.1" 200 191506 "" "Prometheus/2.33.4" 2024-06-01T02:38:51.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:51 smithi146 ceph-mon[25945]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:51.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:51 smithi062 ceph-mon[25095]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:51.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:51 smithi062 ceph-mon[29843]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:51.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:38:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[47864]: level=info ts=2024-06-01T02:38:51.483Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004485637s 2024-06-01T02:38:52.419 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:38:52.420 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "a30646f3-c2f8-42e9-823b-aa4b10416e38", 2024-06-01T02:38:52.420 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "r", 2024-06-01T02:38:52.420 INFO:teuthology.orchestra.run.smithi062.stdout: "current_period": "2313c418-834a-4237-a123-90b18cee9dbf", 2024-06-01T02:38:52.420 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1 2024-06-01T02:38:52.420 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:38:52.705 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:52 smithi062 ceph-mon[25095]: osdmap e61: 8 total, 8 up, 8 in 2024-06-01T02:38:52.705 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:52 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4156383230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-01T02:38:52.705 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:52 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-01T02:38:52.706 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:52 smithi062 ceph-mon[29843]: osdmap e61: 8 total, 8 up, 8 in 2024-06-01T02:38:52.706 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:52 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4156383230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-01T02:38:52.706 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:52 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-01T02:38:52.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:52 smithi146 ceph-mon[25945]: osdmap e61: 8 total, 8 up, 8 in 2024-06-01T02:38:52.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:52 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4156383230' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-01T02:38:52.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:52 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-06-01T02:38:52.943 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-06-01T02:38:53.491 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:53 smithi062 ceph-mon[25095]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:53.491 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:53 smithi062 ceph-mon[25095]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-06-01T02:38:53.491 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:53 smithi062 ceph-mon[25095]: osdmap e62: 8 total, 8 up, 8 in 2024-06-01T02:38:53.492 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:53 smithi062 ceph-mon[29843]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:53.492 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:53 smithi062 ceph-mon[29843]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-06-01T02:38:53.492 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:53 smithi062 ceph-mon[29843]: osdmap e62: 8 total, 8 up, 8 in 2024-06-01T02:38:53.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:53 smithi146 ceph-mon[25945]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:53.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:53 smithi146 ceph-mon[25945]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-06-01T02:38:53.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:53 smithi146 ceph-mon[25945]: osdmap e62: 8 total, 8 up, 8 in 2024-06-01T02:38:53.705 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:53 smithi146 systemd[1]: Starting Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:38:54.205 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:53 smithi146 podman[41930]: 2024-06-01 02:38:53.712878226 +0000 UTC m=+0.017007376 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-06-01T02:38:54.403 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:54 smithi062 ceph-mon[25095]: osdmap e63: 8 total, 8 up, 8 in 2024-06-01T02:38:54.403 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:54 smithi062 ceph-mon[29843]: osdmap e63: 8 total, 8 up, 8 in 2024-06-01T02:38:54.563 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-mon[25945]: osdmap e63: 8 total, 8 up, 8 in 2024-06-01T02:38:54.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 podman[41930]: 2024-06-01 02:38:54.303897536 +0000 UTC m=+0.608026660 container create 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, maintainer=Paul Cuzner , architecture=x86_64, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, version=8.5, distribution-scope=public, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-06-01T02:38:54.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 podman[41930]: 2024-06-01 02:38:54.531104921 +0000 UTC m=+0.835234052 container init 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, vendor=Red Hat, Inc., description=Ceph Grafana Container, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, release=236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, com.redhat.component=ubi8-container, io.openshift.expose-services=) 2024-06-01T02:38:54.564 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 podman[41930]: 2024-06-01 02:38:54.53409756 +0000 UTC m=+0.838226691 container start 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, io.buildah.version=1.24.2, version=8.5, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, description=Ceph Grafana Container, io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, distribution-scope=public, name=ubi8) 2024-06-01T02:38:54.597 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:38:54.597 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "6097f67f-050f-4548-985c-1747f466ea84", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "api_name": "default", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "is_master": "true", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "endpoints": [], 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "hostnames": [], 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "hostnames_s3website": [], 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "master_zone": "", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "zones": [], 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "placement_targets": [], 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "default_placement": "", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "realm_id": "a30646f3-c2f8-42e9-823b-aa4b10416e38", 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_policy": { 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: "groups": [] 2024-06-01T02:38:54.598 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:38:54.599 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:38:54.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-06-01T02:38:54.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-06-01T02:38:54.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-06-01T02:38:54.815 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="App mode production" logger=settings 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-06-01T02:38:54.816 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-06-01T02:38:54.817 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-06-01T02:38:54.817 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-06-01T02:38:54.817 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-06-01T02:38:54.818 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-06-01T02:38:54.819 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-06-01T02:38:54.820 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-06-01T02:38:54.821 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 bash[41930]: 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 2024-06-01T02:38:54.822 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 systemd[1]: Started Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-06-01T02:38:54.823 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-06-01T02:38:54.824 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-06-01T02:38:54.825 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-06-01T02:38:54.825 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-06-01T02:38:54.826 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-06-01T02:38:54.827 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-06-01T02:38:54.828 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-06-01T02:38:54.829 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-06-01T02:38:54.830 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-06-01T02:38:54.831 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-06-01T02:38:54.832 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-06-01T02:38:54.833 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-06-01T02:38:54.834 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-06-01T02:38:54.837 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-06-01T02:38:54.837 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-06-01T02:38:54.837 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-06-01T02:38:54.837 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-06-01T02:38:54.837 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-06-01T02:38:55.081 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-06-01T02:38:55.082 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-06-01T02:38:55.083 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-06-01T02:38:55.083 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-06-01T02:38:55.083 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-06-01T02:38:55.083 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-06-01T02:38:55.084 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-06-01T02:38:55.085 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-06-01T02:38:55.086 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-06-01T02:38:55.087 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-06-01T02:38:55.088 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-06-01T02:38:55.089 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:54 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-06-01T02:38:55.090 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-06-01T02:38:55.091 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-06-01T02:38:55.092 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-06-01T02:38:55.093 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=492.562666ms 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-06-01T02:38:55.094 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-06-01T02:38:55.141 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-06-01T02:38:55.455 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-06-01T02:38:55.455 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-06-01T02:38:55.455 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-06-01T02:38:55.456 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-06-01T02:38:55.456 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-06-01T02:38:55.456 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-06-01T02:38:55.456 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-06-01T02:38:55.456 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:38:55+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-06-01T02:38:56.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[29843]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[25095]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:38:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:55 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:38:56.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-mon[25945]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:38:56.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:56.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:38:56.069 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:55 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:38:56.427 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:38:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:38:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:38:57.556 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:38:57.556 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "7d6f0acc-680d-4963-9066-5153f44489be", 2024-06-01T02:38:57.556 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "z", 2024-06-01T02:38:57.556 INFO:teuthology.orchestra.run.smithi062.stdout: "domain_root": "z.rgw.meta:root", 2024-06-01T02:38:57.556 INFO:teuthology.orchestra.run.smithi062.stdout: "control_pool": "z.rgw.control", 2024-06-01T02:38:57.556 INFO:teuthology.orchestra.run.smithi062.stdout: "gc_pool": "z.rgw.log:gc", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "lc_pool": "z.rgw.log:lc", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "log_pool": "z.rgw.log", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "roles_pool": "z.rgw.meta:roles", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "otp_pool": "z.rgw.otp", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "system_key": { 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "access_key": "", 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: "secret_key": "" 2024-06-01T02:38:57.557 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "placement_pools": [ 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "default-placement", 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "val": { 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "index_pool": "z.rgw.buckets.index", 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "storage_classes": { 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "STANDARD": { 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "data_pool": "z.rgw.buckets.data" 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-06-01T02:38:57.558 INFO:teuthology.orchestra.run.smithi062.stdout: "index_type": 0 2024-06-01T02:38:57.559 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:38:57.559 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:38:57.559 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:38:57.559 INFO:teuthology.orchestra.run.smithi062.stdout: "realm_id": "a30646f3-c2f8-42e9-823b-aa4b10416e38", 2024-06-01T02:38:57.559 INFO:teuthology.orchestra.run.smithi062.stdout: "notif_pool": "z.rgw.log:notif" 2024-06-01T02:38:57.559 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:38:57.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:57 smithi146 ceph-mon[25945]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-06-01T02:38:58.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:57 smithi062 ceph-mon[25095]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-06-01T02:38:58.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:57 smithi062 ceph-mon[29843]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-06-01T02:38:58.285 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-06-01T02:38:59.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:59 smithi062 ceph-mon[25095]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-06-01T02:38:59.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:59 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:59.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:38:59 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:59 smithi062 ceph-mon[29843]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-06-01T02:38:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:59 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:38:59 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:59.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:59 smithi146 ceph-mon[25945]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 628 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-06-01T02:38:59.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:59 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:38:59.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:38:59 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:00.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:39:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:00] "GET /metrics HTTP/1.1" 200 191506 "" "Prometheus/2.33.4" 2024-06-01T02:39:01.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:01 smithi146 ceph-mon[25945]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-06-01T02:39:01.456 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:01 smithi062 ceph-mon[25095]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-06-01T02:39:01.456 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:01 smithi062 ceph-mon[29843]: pgmap v49: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-06-01T02:39:02.351 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[25095]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[25095]: Reconfiguring daemon alertmanager.a on smithi062 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[29843]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-01T02:39:02.352 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:02 smithi062 ceph-mon[29843]: Reconfiguring daemon alertmanager.a on smithi062 2024-06-01T02:39:02.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:02 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:02.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:02 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:02.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:02 smithi146 ceph-mon[25945]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-01T02:39:02.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:02 smithi146 ceph-mon[25945]: Reconfiguring daemon alertmanager.a on smithi062 2024-06-01T02:39:03.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:03 smithi146 ceph-mon[25945]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-06-01T02:39:03.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:03 smithi146 ceph-mon[25945]: osdmap e64: 8 total, 8 up, 8 in 2024-06-01T02:39:03.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:03 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-06-01T02:39:03.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:03 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-06-01T02:39:03.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:03 smithi146 ceph-mon[25945]: osdmap e65: 8 total, 8 up, 8 in 2024-06-01T02:39:03.479 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[25095]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[25095]: osdmap e64: 8 total, 8 up, 8 in 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[25095]: osdmap e65: 8 total, 8 up, 8 in 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[29843]: pgmap v50: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 1.7 KiB/s wr, 2 op/s 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[29843]: osdmap e64: 8 total, 8 up, 8 in 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-06-01T02:39:03.480 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:03 smithi062 ceph-mon[29843]: osdmap e65: 8 total, 8 up, 8 in 2024-06-01T02:39:04.954 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:04 smithi062 systemd[1]: Stopping Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:39:04.954 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:04 smithi062 bash[49540]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager.a" found: no such container 2024-06-01T02:39:04.954 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[47864]: level=info ts=2024-06-01T02:39:04.801Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:39:04.954 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:04 smithi062 podman[49552]: 2024-06-01 02:39:04.822204691 +0000 UTC m=+0.117075668 container died 1e1a4bf9d2324530ec0a1953d0c2caa6372ead8295d4810e8a3dd4a5364a3167 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:04 smithi146 ceph-mon[25945]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 639 B/s wr, 1 op/s 2024-06-01T02:39:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:04 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:04 smithi146 ceph-mon[25945]: osdmap e66: 8 total, 8 up, 8 in 2024-06-01T02:39:05.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:04 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-06-01T02:39:05.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 podman[49552]: 2024-06-01 02:39:05.125779453 +0000 UTC m=+0.420650437 container cleanup 1e1a4bf9d2324530ec0a1953d0c2caa6372ead8295d4810e8a3dd4a5364a3167 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:05.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 bash[49552]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a 2024-06-01T02:39:05.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 bash[49578]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager.a" found: no such container 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[25095]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 639 B/s wr, 1 op/s 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[25095]: osdmap e66: 8 total, 8 up, 8 in 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[29843]: pgmap v53: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 639 B/s wr, 1 op/s 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[29843]: osdmap e66: 8 total, 8 up, 8 in 2024-06-01T02:39:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:04 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-06-01T02:39:05.702 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 podman[49566]: 2024-06-01 02:39:05.377060093 +0000 UTC m=+0.550234600 container remove 1e1a4bf9d2324530ec0a1953d0c2caa6372ead8295d4810e8a3dd4a5364a3167 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:05.702 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 podman[49566]: 2024-06-01 02:39:05.477835877 +0000 UTC m=+0.651010385 volume remove 7b86268c521c505987212521dc0c952a2a3720bbb531f038199f185f2452d2cb 2024-06-01T02:39:05.702 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-01T02:39:05.702 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 systemd[1]: Stopped Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:39:05.702 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 systemd[1]: Starting Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:39:06.012 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 podman[49654]: 2024-06-01 02:39:05.703543336 +0000 UTC m=+0.110504105 volume create 3c7181e228f9d3b4edd3e6da0a6540a2a1d1ded186b90b676a3944777a1bd5f4 2024-06-01T02:39:06.012 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 podman[49654]: 2024-06-01 02:39:05.611227035 +0000 UTC m=+0.018187807 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-06-01T02:39:06.012 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:05 smithi062 podman[49654]: 2024-06-01 02:39:05.762206891 +0000 UTC m=+0.169167651 container create 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:06.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:06 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-06-01T02:39:06.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:06 smithi146 ceph-mon[25945]: osdmap e67: 8 total, 8 up, 8 in 2024-06-01T02:39:06.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:39:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:39:06.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-06-01T02:39:06.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-mon[25095]: osdmap e67: 8 total, 8 up, 8 in 2024-06-01T02:39:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3315408050' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-06-01T02:39:06.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-mon[29843]: osdmap e67: 8 total, 8 up, 8 in 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 podman[49654]: 2024-06-01 02:39:06.115277679 +0000 UTC m=+0.522238432 container init 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 podman[49654]: 2024-06-01 02:39:06.118288688 +0000 UTC m=+0.525249453 container start 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.128Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.128Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.129Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.62 port=9094 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.130Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.156Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.156Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.158Z caller=main.go:518 msg=Listening address=:9093 2024-06-01T02:39:06.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:06.158Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-06-01T02:39:06.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 bash[49654]: 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 2024-06-01T02:39:06.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:06 smithi062 systemd[1]: Started Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:39:07.438 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[25095]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-06-01T02:39:07.438 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[25095]: osdmap e68: 8 total, 8 up, 8 in 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[25095]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[25095]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[29843]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[29843]: osdmap e68: 8 total, 8 up, 8 in 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[29843]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:39:07.439 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:07 smithi062 ceph-mon[29843]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:39:07.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:07 smithi146 ceph-mon[25945]: pgmap v56: 97 pgs: 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-06-01T02:39:07.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:07 smithi146 ceph-mon[25945]: osdmap e68: 8 total, 8 up, 8 in 2024-06-01T02:39:07.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:07 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-06-01T02:39:07.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:07 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:07.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:07 smithi146 ceph-mon[25945]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:39:07.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:07 smithi146 ceph-mon[25945]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:39:08.289 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:08 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-06-01T02:39:08.289 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:08 smithi146 ceph-mon[25945]: osdmap e69: 8 total, 8 up, 8 in 2024-06-01T02:39:08.289 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:08 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-06-01T02:39:08.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-06-01T02:39:08.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-mon[29843]: osdmap e69: 8 total, 8 up, 8 in 2024-06-01T02:39:08.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-06-01T02:39:08.512 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:08.131Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000599805s 2024-06-01T02:39:08.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-06-01T02:39:08.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-mon[25095]: osdmap e69: 8 total, 8 up, 8 in 2024-06-01T02:39:08.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:08 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-06-01T02:39:09.258 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:39:09.258 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "b67f0827-cf0e-4605-ad1d-5da68af41743", 2024-06-01T02:39:09.258 INFO:teuthology.orchestra.run.smithi062.stdout: "epoch": 1, 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "predecessor_uuid": "2313c418-834a-4237-a123-90b18cee9dbf", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_status": [], 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "period_map": { 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "b67f0827-cf0e-4605-ad1d-5da68af41743", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "zonegroups": [ 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "6097f67f-050f-4548-985c-1747f466ea84", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "api_name": "default", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "is_master": "true", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "endpoints": [], 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "hostnames": [], 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "hostnames_s3website": [], 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "master_zone": "7d6f0acc-680d-4963-9066-5153f44489be", 2024-06-01T02:39:09.259 INFO:teuthology.orchestra.run.smithi062.stdout: "zones": [ 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "id": "7d6f0acc-680d-4963-9066-5153f44489be", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "z", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "endpoints": [], 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "log_meta": "false", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "log_data": "false", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "bucket_index_max_shards": 11, 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "read_only": "false", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "tier_type": "", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_from_all": "true", 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_from": [], 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "redirect_zone": "" 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: "placement_targets": [ 2024-06-01T02:39:09.260 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "name": "default-placement", 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "tags": [], 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "storage_classes": [ 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "STANDARD" 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "default_placement": "default-placement", 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "realm_id": "a30646f3-c2f8-42e9-823b-aa4b10416e38", 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "sync_policy": { 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: "groups": [] 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:39:09.261 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "short_zone_ids": [ 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: { 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "key": "7d6f0acc-680d-4963-9066-5153f44489be", 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "val": 4248248925 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: ] 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "master_zonegroup": "6097f67f-050f-4548-985c-1747f466ea84", 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "master_zone": "7d6f0acc-680d-4963-9066-5153f44489be", 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "period_config": { 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "bucket_quota": { 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": false, 2024-06-01T02:39:09.262 INFO:teuthology.orchestra.run.smithi062.stdout: "check_on_raw": false, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_size": -1, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_size_kb": 0, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_objects": -1 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "user_quota": { 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": false, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "check_on_raw": false, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_size": -1, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_size_kb": 0, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_objects": -1 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "user_ratelimit": { 2024-06-01T02:39:09.263 INFO:teuthology.orchestra.run.smithi062.stdout: "max_read_ops": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_write_ops": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_read_bytes": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_write_bytes": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": false 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "bucket_ratelimit": { 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_read_ops": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_write_ops": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_read_bytes": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_write_bytes": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": false 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "anonymous_ratelimit": { 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_read_ops": 0, 2024-06-01T02:39:09.264 INFO:teuthology.orchestra.run.smithi062.stdout: "max_write_ops": 0, 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: "max_read_bytes": 0, 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: "max_write_bytes": 0, 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: "enabled": false 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: "realm_id": "a30646f3-c2f8-42e9-823b-aa4b10416e38", 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: "realm_name": "r", 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout: "realm_epoch": 2 2024-06-01T02:39:09.265 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:39:09.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-mon[25945]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-06-01T02:39:09.322 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-06-01T02:39:09.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-mon[25945]: osdmap e70: 8 total, 8 up, 8 in 2024-06-01T02:39:09.323 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-06-01T02:39:09.323 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 systemd[1]: Stopping Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:39:09.323 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 bash[42620]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:39:09.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[29843]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-06-01T02:39:09.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-06-01T02:39:09.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[29843]: osdmap e70: 8 total, 8 up, 8 in 2024-06-01T02:39:09.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-06-01T02:39:09.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[25095]: pgmap v59: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-06-01T02:39:09.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-06-01T02:39:09.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[25095]: osdmap e70: 8 total, 8 up, 8 in 2024-06-01T02:39:09.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:09 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-06-01T02:39:09.596 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.323Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:39:09.596 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.323Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-01T02:39:09.596 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.323Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-01T02:39:09.596 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.323Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.323Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.323Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.324Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.324Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.324Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.324Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.324Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[40991]: ts=2024-06-01T02:39:09.324Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 podman[42632]: 2024-06-01 02:39:09.334119762 +0000 UTC m=+0.125502181 container died 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 podman[42632]: 2024-06-01 02:39:09.597673988 +0000 UTC m=+0.389056410 container cleanup 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:09.597 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 bash[42632]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a 2024-06-01T02:39:09.856 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 bash[42659]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:39:10.090 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-06-01T02:39:10.115 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 podman[42645]: 2024-06-01 02:39:09.857388095 +0000 UTC m=+0.524882581 container remove 63eb86c593d3686c0ec58080dfb26b112ace6790a8455c2afde27a62fcd740d7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:10.115 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-01T02:39:10.115 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 systemd[1]: Stopped Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:39:10.115 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:09 smithi146 systemd[1]: Starting Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:39:10.115 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 podman[42718]: 2024-06-01 02:39:09.995557844 +0000 UTC m=+0.018596003 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-01T02:39:10.115 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 podman[42718]: 2024-06-01 02:39:10.116663184 +0000 UTC m=+0.139701376 container create ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:10.372 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-06-01T02:39:10.372 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-mon[25945]: osdmap e71: 8 total, 8 up, 8 in 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 podman[42718]: 2024-06-01 02:39:10.293712682 +0000 UTC m=+0.316750832 container init ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 podman[42718]: 2024-06-01 02:39:10.296672337 +0000 UTC m=+0.319710493 container start ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.319Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.319Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.319Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.319Z caller=main.go:518 level=info host_details="(Linux 5.14.0-452.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024 x86_64 smithi146 (none))" 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.319Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.319Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.321Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.322Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.323Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-01T02:39:10.373 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.325Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-01T02:39:10.374 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.325Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.526µs 2024-06-01T02:39:10.374 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.325Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-01T02:39:10.374 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 bash[42718]: ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff 2024-06-01T02:39:10.374 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 systemd[1]: Started Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:39:10.415 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:10 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-06-01T02:39:10.415 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:10 smithi062 ceph-mon[25095]: osdmap e71: 8 total, 8 up, 8 in 2024-06-01T02:39:10.415 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:10 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1500411520' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-06-01T02:39:10.415 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:10 smithi062 ceph-mon[29843]: osdmap e71: 8 total, 8 up, 8 in 2024-06-01T02:39:10.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.517Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-06-01T02:39:10.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.517Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-06-01T02:39:10.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.517Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=32.343µs wal_replay_duration=192.299692ms total_replay_duration=192.345701ms 2024-06-01T02:39:10.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.519Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-01T02:39:10.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.519Z caller=main.go:947 level=info msg="TSDB started" 2024-06-01T02:39:10.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.519Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-01T02:39:10.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.537Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.702563ms db_storage=749ns remote_storage=1.506µs web_handler=346ns query_engine=646ns scrape=235.486µs scrape_sd=24.396µs notify=27.837µs notify_sd=12.112µs rules=17.098402ms 2024-06-01T02:39:10.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:39:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:39:10.537Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-01T02:39:11.475 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-06-01T02:39:11.475 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.62:9093"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.62:9093"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.146:9095"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.146:9095"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:39:11.476 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.146:3000"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.146:3000"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:39:11.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.62:9093"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.62:9093"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.146:9095"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.146:9095"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.146:3000"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.146:3000"}]: dispatch 2024-06-01T02:39:11.478 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.479 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:11.479 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:11 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: pgmap v62: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-06-01T02:39:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:39:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.62:9093"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.62:9093"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.146:9095"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.146:9095"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.146:3000"}]: dispatch 2024-06-01T02:39:11.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.146:3000"}]: dispatch 2024-06-01T02:39:11.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:11.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:11.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:11 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:12.279 INFO:teuthology.orchestra.run.smithi062.stdout:Scheduled rgw.foo update... 2024-06-01T02:39:13.085 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch apply rgw smpl' 2024-06-01T02:39:13.414 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[25095]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[25095]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[25095]: Saving service rgw.foo spec with placement count:2 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[29843]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[29843]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[29843]: Saving service rgw.foo spec with placement count:2 2024-06-01T02:39:13.415 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:13 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:13.554 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:13 smithi146 ceph-mon[25945]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-06-01T02:39:13.554 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:13 smithi146 ceph-mon[25945]: from='client.24577 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:13.554 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:13 smithi146 ceph-mon[25945]: Saving service rgw.foo spec with placement count:2 2024-06-01T02:39:13.554 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:13 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:14.921 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:14 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:14.922 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:14 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:15.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:14 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:15.235 INFO:teuthology.orchestra.run.smithi062.stdout:Scheduled rgw.smpl update... 2024-06-01T02:39:15.986 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:15 smithi062 ceph-mon[29843]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-06-01T02:39:15.986 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:15 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:15.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:15 smithi062 ceph-mon[25095]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-06-01T02:39:15.986 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:15 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:16.013 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph osd pool create foo' 2024-06-01T02:39:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:15 smithi146 ceph-mon[25945]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-06-01T02:39:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:15 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:16.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:39:16.134Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.003628623s 2024-06-01T02:39:16.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:39:17.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:17.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: Saving service rgw.smpl spec with placement count:2 2024-06-01T02:39:17.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: Saving service rgw.smpl spec with placement count:2 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:17.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:17.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:16 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='client.24583 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: Saving service rgw.smpl spec with placement count:2 2024-06-01T02:39:17.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:16 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:18.059 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[25095]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-06-01T02:39:18.059 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[25095]: Saving service rgw.foo spec with placement count:2 2024-06-01T02:39:18.059 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:18.059 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:18.060 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[29843]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-06-01T02:39:18.060 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[29843]: Saving service rgw.foo spec with placement count:2 2024-06-01T02:39:18.060 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:18.060 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:17 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:17 smithi146 ceph-mon[25945]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-06-01T02:39:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:17 smithi146 ceph-mon[25945]: Saving service rgw.foo spec with placement count:2 2024-06-01T02:39:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:17 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:17 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:19.152 INFO:teuthology.orchestra.run.smithi062.stderr:pool 'foo' created 2024-06-01T02:39:19.442 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:19 smithi062 ceph-mon[25095]: Deploying daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:39:19.443 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:19 smithi062 ceph-mon[29843]: Deploying daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:39:19.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:19 smithi146 ceph-mon[25945]: Deploying daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:39:19.881 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'rbd pool init foo' 2024-06-01T02:39:20.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[25095]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 950 B/s wr, 2 op/s 2024-06-01T02:39:20.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2631752560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-01T02:39:20.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2631752560' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-06-01T02:39:20.207 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[25095]: osdmap e72: 8 total, 8 up, 8 in 2024-06-01T02:39:20.208 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[29843]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 950 B/s wr, 2 op/s 2024-06-01T02:39:20.208 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2631752560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-01T02:39:20.208 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2631752560' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-06-01T02:39:20.208 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-mon[29843]: osdmap e72: 8 total, 8 up, 8 in 2024-06-01T02:39:20.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:20 smithi146 ceph-mon[25945]: pgmap v66: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 844 B/s rd, 950 B/s wr, 2 op/s 2024-06-01T02:39:20.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:20 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2631752560' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-06-01T02:39:20.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:20 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2631752560' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-06-01T02:39:20.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:20 smithi146 ceph-mon[25945]: osdmap e72: 8 total, 8 up, 8 in 2024-06-01T02:39:20.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:39:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:20] "GET /metrics HTTP/1.1" 200 196826 "" "Prometheus/2.33.4" 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: pgmap v68: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: osdmap e73: 8 total, 8 up, 8 in 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:21.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[29843]: Deploying daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: pgmap v68: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: osdmap e73: 8 total, 8 up, 8 in 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:21.427 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:21 smithi062 ceph-mon[25095]: Deploying daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: pgmap v68: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: osdmap e73: 8 total, 8 up, 8 in 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:21.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:21.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:21 smithi146 ceph-mon[25945]: Deploying daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:39:22.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:22 smithi062 ceph-mon[25095]: osdmap e74: 8 total, 8 up, 8 in 2024-06-01T02:39:22.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:22 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1131459874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-06-01T02:39:22.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:22 smithi062 ceph-mon[29843]: osdmap e74: 8 total, 8 up, 8 in 2024-06-01T02:39:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:22 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1131459874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-06-01T02:39:22.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:22 smithi146 ceph-mon[25945]: osdmap e74: 8 total, 8 up, 8 in 2024-06-01T02:39:22.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:22 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1131459874' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-06-01T02:39:23.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:23 smithi146 ceph-mon[25945]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:23.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:23 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1131459874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-06-01T02:39:23.425 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:23 smithi146 ceph-mon[25945]: osdmap e75: 8 total, 8 up, 8 in 2024-06-01T02:39:23.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:23 smithi062 ceph-mon[25095]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:23.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:23 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1131459874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-06-01T02:39:23.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:23 smithi062 ceph-mon[25095]: osdmap e75: 8 total, 8 up, 8 in 2024-06-01T02:39:23.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:23 smithi062 ceph-mon[29843]: pgmap v71: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:23.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:23 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1131459874' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-06-01T02:39:23.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:23 smithi062 ceph-mon[29843]: osdmap e75: 8 total, 8 up, 8 in 2024-06-01T02:39:24.444 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:24 smithi062 ceph-mon[25095]: osdmap e76: 8 total, 8 up, 8 in 2024-06-01T02:39:24.445 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:24 smithi062 ceph-mon[29843]: osdmap e76: 8 total, 8 up, 8 in 2024-06-01T02:39:24.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:24 smithi146 ceph-mon[25945]: osdmap e76: 8 total, 8 up, 8 in 2024-06-01T02:39:24.877 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch apply iscsi foo u p' 2024-06-01T02:39:25.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:25 smithi146 ceph-mon[25945]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:25.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:25 smithi146 ceph-mon[25945]: osdmap e77: 8 total, 8 up, 8 in 2024-06-01T02:39:25.432 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:25 smithi062 ceph-mon[25095]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:25.432 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:25 smithi062 ceph-mon[25095]: osdmap e77: 8 total, 8 up, 8 in 2024-06-01T02:39:25.432 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:25 smithi062 ceph-mon[29843]: pgmap v74: 161 pgs: 32 creating+peering, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:25.432 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:25 smithi062 ceph-mon[29843]: osdmap e77: 8 total, 8 up, 8 in 2024-06-01T02:39:26.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:39:26.843 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:26.843 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: Saving service rgw.smpl spec with placement count:2 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[25095]: Deploying daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: Saving service rgw.smpl spec with placement count:2 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:26.844 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:26.845 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:26.845 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:26.845 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:26.845 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:26 smithi062 ceph-mon[29843]: Deploying daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:39:27.053 INFO:teuthology.orchestra.run.smithi062.stdout:Scheduled iscsi.foo update... 2024-06-01T02:39:27.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:27.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: Saving service rgw.smpl spec with placement count:2 2024-06-01T02:39:27.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:27.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:26 smithi146 ceph-mon[25945]: Deploying daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:39:27.744 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'sleep 120' 2024-06-01T02:39:28.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:27 smithi062 ceph-mon[25095]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 7.4 KiB/s wr, 330 op/s 2024-06-01T02:39:28.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:27 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:28.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:27 smithi062 ceph-mon[29843]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 7.4 KiB/s wr, 330 op/s 2024-06-01T02:39:28.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:27 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:27 smithi146 ceph-mon[25945]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 154 KiB/s rd, 7.4 KiB/s wr, 330 op/s 2024-06-01T02:39:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:27 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:29.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:28 smithi062 ceph-mon[25095]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:29.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:28 smithi062 ceph-mon[25095]: Saving service iscsi.foo spec with placement count:1 2024-06-01T02:39:29.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:28 smithi062 ceph-mon[29843]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:29.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:28 smithi062 ceph-mon[29843]: Saving service iscsi.foo spec with placement count:1 2024-06-01T02:39:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:28 smithi146 ceph-mon[25945]: from='client.24637 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:39:29.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:28 smithi146 ceph-mon[25945]: Saving service iscsi.foo spec with placement count:1 2024-06-01T02:39:30.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:29 smithi062 ceph-mon[25095]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 5.8 KiB/s wr, 261 op/s 2024-06-01T02:39:30.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:29 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:30.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:29 smithi062 ceph-mon[25095]: osdmap e78: 8 total, 8 up, 8 in 2024-06-01T02:39:30.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:29 smithi062 ceph-mon[29843]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 5.8 KiB/s wr, 261 op/s 2024-06-01T02:39:30.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:29 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:30.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:29 smithi062 ceph-mon[29843]: osdmap e78: 8 total, 8 up, 8 in 2024-06-01T02:39:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:29 smithi146 ceph-mon[25945]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s rd, 5.8 KiB/s wr, 261 op/s 2024-06-01T02:39:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:29 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:29 smithi146 ceph-mon[25945]: osdmap e78: 8 total, 8 up, 8 in 2024-06-01T02:39:30.779 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:39:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:30] "GET /metrics HTTP/1.1" 200 196826 "" "Prometheus/2.33.4" 2024-06-01T02:39:31.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 235 KiB/s rd, 5.4 KiB/s wr, 428 op/s 2024-06-01T02:39:31.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:31.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:31.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:31.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:31.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:31.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:31 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 235 KiB/s rd, 5.4 KiB/s wr, 428 op/s 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 235 KiB/s rd, 5.4 KiB/s wr, 428 op/s 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:31 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:32.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:32 smithi146 ceph-mon[25945]: Deploying daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:39:33.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:32 smithi062 ceph-mon[25095]: Deploying daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:39:33.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:32 smithi062 ceph-mon[29843]: Deploying daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:39:34.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[25095]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.5 KiB/s wr, 360 op/s 2024-06-01T02:39:34.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:34.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:34.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[29843]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.5 KiB/s wr, 360 op/s 2024-06-01T02:39:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:34.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:34.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:33 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:34.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:33 smithi146 ceph-mon[25945]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 198 KiB/s rd, 4.5 KiB/s wr, 360 op/s 2024-06-01T02:39:34.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:33 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:34.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:33 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:34.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:33 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:35.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-06-01T02:39:35.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:39:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:39:35.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:39:35.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:35.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-06-01T02:39:35.076 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:39:35.077 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:36.086 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[25095]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 3.7 KiB/s wr, 296 op/s 2024-06-01T02:39:36.087 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-06-01T02:39:36.087 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-06-01T02:39:36.087 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-06-01T02:39:36.087 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-01T02:39:36.087 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[25095]: osdmap e79: 8 total, 8 up, 8 in 2024-06-01T02:39:36.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[29843]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 3.7 KiB/s wr, 296 op/s 2024-06-01T02:39:36.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-06-01T02:39:36.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-06-01T02:39:36.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-06-01T02:39:36.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-01T02:39:36.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:35 smithi062 ceph-mon[29843]: osdmap e79: 8 total, 8 up, 8 in 2024-06-01T02:39:36.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:35 smithi146 ceph-mon[25945]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 65 MiB used, 715 GiB / 715 GiB avail; 163 KiB/s rd, 3.7 KiB/s wr, 296 op/s 2024-06-01T02:39:36.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [7, 2]}]': finished 2024-06-01T02:39:36.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-06-01T02:39:36.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 5]}]': finished 2024-06-01T02:39:36.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:35 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1d", "id": [1, 2]}]': finished 2024-06-01T02:39:36.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:35 smithi146 ceph-mon[25945]: osdmap e79: 8 total, 8 up, 8 in 2024-06-01T02:39:36.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:39:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:39:36.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:36 smithi146 ceph-mon[25945]: osdmap e80: 8 total, 8 up, 8 in 2024-06-01T02:39:36.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:36 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:37.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:36 smithi062 ceph-mon[25095]: osdmap e80: 8 total, 8 up, 8 in 2024-06-01T02:39:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:36 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:36 smithi062 ceph-mon[29843]: osdmap e80: 8 total, 8 up, 8 in 2024-06-01T02:39:37.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:36 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.004 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 443 KiB/s rd, 451 B/s wr, 677 op/s 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 443 KiB/s rd, 451 B/s wr, 677 op/s 2024-06-01T02:39:38.005 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-06-01T02:39:38.006 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:37 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:38.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 443 KiB/s rd, 451 B/s wr, 677 op/s 2024-06-01T02:39:38.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:39:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:39:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-06-01T02:39:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:37 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:39.152 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:38 smithi062 ceph-mon[25095]: Checking dashboard <-> RGW credentials 2024-06-01T02:39:39.152 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:38 smithi062 ceph-mon[25095]: Checking pool "foo" exists for service iscsi.foo 2024-06-01T02:39:39.152 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:38 smithi062 ceph-mon[25095]: Deploying daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:39:39.152 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:38 smithi062 ceph-mon[29843]: Checking dashboard <-> RGW credentials 2024-06-01T02:39:39.153 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:38 smithi062 ceph-mon[29843]: Checking pool "foo" exists for service iscsi.foo 2024-06-01T02:39:39.153 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:38 smithi062 ceph-mon[29843]: Deploying daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:39:39.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:38 smithi146 ceph-mon[25945]: Checking dashboard <-> RGW credentials 2024-06-01T02:39:39.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:38 smithi146 ceph-mon[25945]: Checking pool "foo" exists for service iscsi.foo 2024-06-01T02:39:39.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:38 smithi146 ceph-mon[25945]: Deploying daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:39:39.856 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:39 smithi062 ceph-mon[25095]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 270 KiB/s rd, 255 B/s wr, 411 op/s 2024-06-01T02:39:39.857 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:39 smithi062 ceph-mon[29843]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 270 KiB/s rd, 255 B/s wr, 411 op/s 2024-06-01T02:39:40.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:39 smithi146 ceph-mon[25945]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 270 KiB/s rd, 255 B/s wr, 411 op/s 2024-06-01T02:39:41.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:39:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:40] "GET /metrics HTTP/1.1" 200 213364 "" "Prometheus/2.33.4" 2024-06-01T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:41 smithi146 ceph-mon[25945]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 296 KiB/s rd, 511 B/s wr, 449 op/s 2024-06-01T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:41 smithi146 ceph-mon[25945]: mgrmap e21: y(active, since 2m), standbys: x 2024-06-01T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:41 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:41 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:41.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:41 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3142559426' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:39:41.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[25095]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 296 KiB/s rd, 511 B/s wr, 449 op/s 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[25095]: mgrmap e21: y(active, since 2m), standbys: x 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3142559426' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[29843]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 296 KiB/s rd, 511 B/s wr, 449 op/s 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[29843]: mgrmap e21: y(active, since 2m), standbys: x 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:41.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:41.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:41 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3142559426' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:39:42.325 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:42 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2393844832' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/779526503"}]: dispatch 2024-06-01T02:39:42.326 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:42 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2393844832' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/779526503"}]: dispatch 2024-06-01T02:39:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:42 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2393844832' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/779526503"}]: dispatch 2024-06-01T02:39:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:43 smithi146 ceph-mon[25945]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 296 KiB/s rd, 511 B/s wr, 449 op/s 2024-06-01T02:39:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:43 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2393844832' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/779526503"}]': finished 2024-06-01T02:39:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:43 smithi146 ceph-mon[25945]: osdmap e81: 8 total, 8 up, 8 in 2024-06-01T02:39:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:43 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2785833677' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]: dispatch 2024-06-01T02:39:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:43 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]: dispatch 2024-06-01T02:39:43.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[29843]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 296 KiB/s rd, 511 B/s wr, 449 op/s 2024-06-01T02:39:43.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2393844832' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/779526503"}]': finished 2024-06-01T02:39:43.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[29843]: osdmap e81: 8 total, 8 up, 8 in 2024-06-01T02:39:43.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2785833677' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]: dispatch 2024-06-01T02:39:43.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]: dispatch 2024-06-01T02:39:43.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[25095]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 296 KiB/s rd, 511 B/s wr, 449 op/s 2024-06-01T02:39:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2393844832' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/779526503"}]': finished 2024-06-01T02:39:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[25095]: osdmap e81: 8 total, 8 up, 8 in 2024-06-01T02:39:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2785833677' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]: dispatch 2024-06-01T02:39:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]: dispatch 2024-06-01T02:39:43.981 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:39:43.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:39:43.981 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:39:43.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]': finished 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: osdmap e82: 8 total, 8 up, 8 in 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1714794479' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]: dispatch 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]: dispatch 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-01T02:39:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi062"}]: dispatch 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]': finished 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: osdmap e82: 8 total, 8 up, 8 in 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1714794479' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]: dispatch 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]: dispatch 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi062"}]: dispatch 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:44.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:44 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:44.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1986861472"}]': finished 2024-06-01T02:39:44.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: osdmap e82: 8 total, 8 up, 8 in 2024-06-01T02:39:44.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1714794479' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]: dispatch 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]: dispatch 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi062"}]: dispatch 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:39:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:44 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:39:45.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 255 B/s wr, 37 op/s 2024-06-01T02:39:45.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-01T02:39:45.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: Adding iSCSI gateway http://:@172.21.15.62:5000 to Dashboard 2024-06-01T02:39:45.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-01T02:39:45.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi062"}]: dispatch 2024-06-01T02:39:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]': finished 2024-06-01T02:39:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: osdmap e83: 8 total, 8 up, 8 in 2024-06-01T02:39:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3234344239' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]: dispatch 2024-06-01T02:39:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:45 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]: dispatch 2024-06-01T02:39:45.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 255 B/s wr, 37 op/s 2024-06-01T02:39:45.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-01T02:39:45.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: Adding iSCSI gateway http://:@172.21.15.62:5000 to Dashboard 2024-06-01T02:39:45.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-01T02:39:45.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi062"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]': finished 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: osdmap e83: 8 total, 8 up, 8 in 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3234344239' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 255 B/s wr, 37 op/s 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: Adding iSCSI gateway http://:@172.21.15.62:5000 to Dashboard 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi062"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/779526503"}]': finished 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: osdmap e83: 8 total, 8 up, 8 in 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3234344239' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]: dispatch 2024-06-01T02:39:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:45 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]: dispatch 2024-06-01T02:39:46.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:46 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]': finished 2024-06-01T02:39:46.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:46 smithi062 ceph-mon[25095]: osdmap e84: 8 total, 8 up, 8 in 2024-06-01T02:39:46.409 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:46 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1084468823' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1920187738"}]: dispatch 2024-06-01T02:39:46.409 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:46 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]': finished 2024-06-01T02:39:46.410 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:46 smithi062 ceph-mon[29843]: osdmap e84: 8 total, 8 up, 8 in 2024-06-01T02:39:46.410 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:46 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1084468823' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1920187738"}]: dispatch 2024-06-01T02:39:46.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:46 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1694477920"}]': finished 2024-06-01T02:39:46.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:46 smithi146 ceph-mon[25945]: osdmap e84: 8 total, 8 up, 8 in 2024-06-01T02:39:46.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:46 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1084468823' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1920187738"}]: dispatch 2024-06-01T02:39:46.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:39:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:39:47.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:47 smithi146 ceph-mon[25945]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-06-01T02:39:47.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:47 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1084468823' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1920187738"}]': finished 2024-06-01T02:39:47.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:47 smithi146 ceph-mon[25945]: osdmap e85: 8 total, 8 up, 8 in 2024-06-01T02:39:47.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:47 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/367381444' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/3339555501"}]: dispatch 2024-06-01T02:39:47.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[29843]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-06-01T02:39:47.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1084468823' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1920187738"}]': finished 2024-06-01T02:39:47.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[29843]: osdmap e85: 8 total, 8 up, 8 in 2024-06-01T02:39:47.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/367381444' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/3339555501"}]: dispatch 2024-06-01T02:39:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[25095]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-06-01T02:39:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1084468823' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1920187738"}]': finished 2024-06-01T02:39:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[25095]: osdmap e85: 8 total, 8 up, 8 in 2024-06-01T02:39:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:47 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/367381444' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/3339555501"}]: dispatch 2024-06-01T02:39:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/367381444' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/3339555501"}]': finished 2024-06-01T02:39:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: osdmap e86: 8 total, 8 up, 8 in 2024-06-01T02:39:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/983163364' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]: dispatch 2024-06-01T02:39:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]: dispatch 2024-06-01T02:39:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: Checking dashboard <-> RGW credentials 2024-06-01T02:39:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:48 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/367381444' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/3339555501"}]': finished 2024-06-01T02:39:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: osdmap e86: 8 total, 8 up, 8 in 2024-06-01T02:39:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/983163364' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]: dispatch 2024-06-01T02:39:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]: dispatch 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: Checking dashboard <-> RGW credentials 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/367381444' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/3339555501"}]': finished 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: osdmap e86: 8 total, 8 up, 8 in 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/983163364' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]: dispatch 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]: dispatch 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:48.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: Checking dashboard <-> RGW credentials 2024-06-01T02:39:48.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:48 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:49.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:49 smithi146 ceph-mon[25945]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-06-01T02:39:49.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:49 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]': finished 2024-06-01T02:39:49.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:49 smithi146 ceph-mon[25945]: osdmap e87: 8 total, 8 up, 8 in 2024-06-01T02:39:49.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:49 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4292516016' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3491871"}]: dispatch 2024-06-01T02:39:49.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:49 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:49.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[29843]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-06-01T02:39:49.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]': finished 2024-06-01T02:39:49.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[29843]: osdmap e87: 8 total, 8 up, 8 in 2024-06-01T02:39:49.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4292516016' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3491871"}]: dispatch 2024-06-01T02:39:49.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[25095]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-06-01T02:39:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4106139753"}]': finished 2024-06-01T02:39:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[25095]: osdmap e87: 8 total, 8 up, 8 in 2024-06-01T02:39:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4292516016' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3491871"}]: dispatch 2024-06-01T02:39:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:49 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:39:50.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:50 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4292516016' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3491871"}]': finished 2024-06-01T02:39:50.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:50 smithi146 ceph-mon[25945]: osdmap e88: 8 total, 8 up, 8 in 2024-06-01T02:39:50.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:50 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2587492539' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]: dispatch 2024-06-01T02:39:50.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:50 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]: dispatch 2024-06-01T02:39:50.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4292516016' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3491871"}]': finished 2024-06-01T02:39:50.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[25095]: osdmap e88: 8 total, 8 up, 8 in 2024-06-01T02:39:50.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2587492539' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]: dispatch 2024-06-01T02:39:50.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]: dispatch 2024-06-01T02:39:50.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4292516016' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3491871"}]': finished 2024-06-01T02:39:50.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[29843]: osdmap e88: 8 total, 8 up, 8 in 2024-06-01T02:39:50.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2587492539' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]: dispatch 2024-06-01T02:39:50.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]: dispatch 2024-06-01T02:39:51.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:39:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:50] "GET /metrics HTTP/1.1" 200 214950 "" "Prometheus/2.33.4" 2024-06-01T02:39:51.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:51 smithi146 ceph-mon[25945]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-06-01T02:39:51.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:51 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]': finished 2024-06-01T02:39:51.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:51 smithi146 ceph-mon[25945]: osdmap e89: 8 total, 8 up, 8 in 2024-06-01T02:39:51.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:51 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3872959584' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]: dispatch 2024-06-01T02:39:51.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:51 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]: dispatch 2024-06-01T02:39:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:51 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:39:51.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[25095]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-06-01T02:39:51.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]': finished 2024-06-01T02:39:51.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[25095]: osdmap e89: 8 total, 8 up, 8 in 2024-06-01T02:39:51.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3872959584' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]: dispatch 2024-06-01T02:39:51.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]: dispatch 2024-06-01T02:39:51.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:39:51.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[29843]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-06-01T02:39:51.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1842014486"}]': finished 2024-06-01T02:39:51.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[29843]: osdmap e89: 8 total, 8 up, 8 in 2024-06-01T02:39:51.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3872959584' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]: dispatch 2024-06-01T02:39:51.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]: dispatch 2024-06-01T02:39:51.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:51 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:39:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:52 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]': finished 2024-06-01T02:39:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:52 smithi146 ceph-mon[25945]: osdmap e90: 8 total, 8 up, 8 in 2024-06-01T02:39:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:52 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1210433086' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]: dispatch 2024-06-01T02:39:52.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:52 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]: dispatch 2024-06-01T02:39:52.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]': finished 2024-06-01T02:39:52.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[29843]: osdmap e90: 8 total, 8 up, 8 in 2024-06-01T02:39:52.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1210433086' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]: dispatch 2024-06-01T02:39:52.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]: dispatch 2024-06-01T02:39:52.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3141512320"}]': finished 2024-06-01T02:39:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[25095]: osdmap e90: 8 total, 8 up, 8 in 2024-06-01T02:39:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1210433086' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]: dispatch 2024-06-01T02:39:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:52 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]: dispatch 2024-06-01T02:39:53.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:53 smithi146 ceph-mon[25945]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-06-01T02:39:53.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:53 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]': finished 2024-06-01T02:39:53.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:53 smithi146 ceph-mon[25945]: osdmap e91: 8 total, 8 up, 8 in 2024-06-01T02:39:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:53 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3395115815' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]: dispatch 2024-06-01T02:39:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:53 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]: dispatch 2024-06-01T02:39:53.508 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[25095]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-06-01T02:39:53.508 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]': finished 2024-06-01T02:39:53.508 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[25095]: osdmap e91: 8 total, 8 up, 8 in 2024-06-01T02:39:53.508 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3395115815' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]: dispatch 2024-06-01T02:39:53.508 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]: dispatch 2024-06-01T02:39:53.509 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[29843]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-06-01T02:39:53.509 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1434776916"}]': finished 2024-06-01T02:39:53.509 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[29843]: osdmap e91: 8 total, 8 up, 8 in 2024-06-01T02:39:53.509 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3395115815' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]: dispatch 2024-06-01T02:39:53.509 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]: dispatch 2024-06-01T02:39:53.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:39:53.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:39:53.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:39:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:39:53.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:39:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:39:53.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:39:54.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:54 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]': finished 2024-06-01T02:39:54.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:54 smithi146 ceph-mon[25945]: osdmap e92: 8 total, 8 up, 8 in 2024-06-01T02:39:54.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:54 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/823397834' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1434776916"}]: dispatch 2024-06-01T02:39:54.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:54 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]': finished 2024-06-01T02:39:54.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:54 smithi062 ceph-mon[25095]: osdmap e92: 8 total, 8 up, 8 in 2024-06-01T02:39:54.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:54 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/823397834' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1434776916"}]: dispatch 2024-06-01T02:39:54.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:54 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1726234481"}]': finished 2024-06-01T02:39:54.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:54 smithi062 ceph-mon[29843]: osdmap e92: 8 total, 8 up, 8 in 2024-06-01T02:39:54.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:54 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/823397834' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1434776916"}]: dispatch 2024-06-01T02:39:55.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:55 smithi146 ceph-mon[25945]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:55.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:55 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/823397834' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1434776916"}]': finished 2024-06-01T02:39:55.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:55 smithi146 ceph-mon[25945]: osdmap e93: 8 total, 8 up, 8 in 2024-06-01T02:39:55.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:55 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/526464744' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2403150514"}]: dispatch 2024-06-01T02:39:55.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[25095]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:55.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/823397834' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1434776916"}]': finished 2024-06-01T02:39:55.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[25095]: osdmap e93: 8 total, 8 up, 8 in 2024-06-01T02:39:55.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/526464744' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2403150514"}]: dispatch 2024-06-01T02:39:55.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[29843]: pgmap v105: 161 pgs: 161 active+clean; 585 KiB data, 95 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:39:55.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/823397834' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1434776916"}]': finished 2024-06-01T02:39:55.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[29843]: osdmap e93: 8 total, 8 up, 8 in 2024-06-01T02:39:55.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:55 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/526464744' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2403150514"}]: dispatch 2024-06-01T02:39:56.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:56 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/526464744' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2403150514"}]': finished 2024-06-01T02:39:56.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:56 smithi146 ceph-mon[25945]: osdmap e94: 8 total, 8 up, 8 in 2024-06-01T02:39:56.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:56 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/41841470' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]: dispatch 2024-06-01T02:39:56.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:56 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]: dispatch 2024-06-01T02:39:56.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:39:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:39:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:39:56.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/526464744' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2403150514"}]': finished 2024-06-01T02:39:56.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[29843]: osdmap e94: 8 total, 8 up, 8 in 2024-06-01T02:39:56.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/41841470' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]: dispatch 2024-06-01T02:39:56.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]: dispatch 2024-06-01T02:39:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/526464744' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2403150514"}]': finished 2024-06-01T02:39:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[25095]: osdmap e94: 8 total, 8 up, 8 in 2024-06-01T02:39:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/41841470' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]: dispatch 2024-06-01T02:39:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:56 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]: dispatch 2024-06-01T02:39:57.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[29843]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:39:57.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]': finished 2024-06-01T02:39:57.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[29843]: osdmap e95: 8 total, 8 up, 8 in 2024-06-01T02:39:57.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3317090953' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3889207289"}]: dispatch 2024-06-01T02:39:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[25095]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:39:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]': finished 2024-06-01T02:39:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[25095]: osdmap e95: 8 total, 8 up, 8 in 2024-06-01T02:39:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:57 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3317090953' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3889207289"}]: dispatch 2024-06-01T02:39:57.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:57 smithi146 ceph-mon[25945]: pgmap v108: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:39:57.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:57 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/3339555501"}]': finished 2024-06-01T02:39:57.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:57 smithi146 ceph-mon[25945]: osdmap e95: 8 total, 8 up, 8 in 2024-06-01T02:39:57.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:57 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3317090953' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3889207289"}]: dispatch 2024-06-01T02:39:58.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:58 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3317090953' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3889207289"}]': finished 2024-06-01T02:39:58.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:58 smithi062 ceph-mon[25095]: osdmap e96: 8 total, 8 up, 8 in 2024-06-01T02:39:58.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:58 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3317090953' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3889207289"}]': finished 2024-06-01T02:39:58.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:58 smithi062 ceph-mon[29843]: osdmap e96: 8 total, 8 up, 8 in 2024-06-01T02:39:58.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:58 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3317090953' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3889207289"}]': finished 2024-06-01T02:39:58.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:58 smithi146 ceph-mon[25945]: osdmap e96: 8 total, 8 up, 8 in 2024-06-01T02:39:59.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:39:59 smithi062 ceph-mon[25095]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:39:59.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:39:59 smithi062 ceph-mon[29843]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:39:59.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:39:59 smithi146 ceph-mon[25945]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:00.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:00 smithi062 ceph-mon[25095]: overall HEALTH_OK 2024-06-01T02:40:00.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:00 smithi062 ceph-mon[29843]: overall HEALTH_OK 2024-06-01T02:40:00.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:00 smithi146 ceph-mon[25945]: overall HEALTH_OK 2024-06-01T02:40:01.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:40:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:00] "GET /metrics HTTP/1.1" 200 214950 "" "Prometheus/2.33.4" 2024-06-01T02:40:01.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:01 smithi062 ceph-mon[25095]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-06-01T02:40:01.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:01 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:01.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:01 smithi062 ceph-mon[29843]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-06-01T02:40:01.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:01 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:01.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:01 smithi146 ceph-mon[25945]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1 op/s 2024-06-01T02:40:01.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:01 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:03.509 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:03 smithi062 ceph-mon[25095]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 0 op/s 2024-06-01T02:40:03.509 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:03 smithi062 ceph-mon[29843]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 0 op/s 2024-06-01T02:40:03.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:03 smithi146 ceph-mon[25945]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 763 B/s rd, 0 op/s 2024-06-01T02:40:03.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:40:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:40:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:40:03.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:40:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:40:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:40:03.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:40:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:40:03.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:40:05.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:05 smithi062 ceph-mon[25095]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-01T02:40:05.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:05 smithi062 ceph-mon[29843]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-01T02:40:05.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:05 smithi146 ceph-mon[25945]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-01T02:40:06.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:40:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:40:07.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:07 smithi062 ceph-mon[25095]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-01T02:40:07.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:07 smithi062 ceph-mon[29843]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-01T02:40:07.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:07 smithi146 ceph-mon[25945]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-01T02:40:09.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:09 smithi062 ceph-mon[25095]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-06-01T02:40:09.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:09 smithi062 ceph-mon[29843]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-06-01T02:40:09.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:09 smithi146 ceph-mon[25945]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 957 B/s rd, 0 op/s 2024-06-01T02:40:11.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:40:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:10] "GET /metrics HTTP/1.1" 200 214981 "" "Prometheus/2.33.4" 2024-06-01T02:40:11.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:11 smithi062 ceph-mon[25095]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:11.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:11 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:11 smithi062 ceph-mon[29843]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:11 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:11 smithi146 ceph-mon[25945]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:11 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:13.509 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:13 smithi062 ceph-mon[25095]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:13.510 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:13 smithi062 ceph-mon[29843]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:13.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:13 smithi146 ceph-mon[25945]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:13.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:40:13 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:40:13.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:40:15.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:15 smithi062 ceph-mon[25095]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:15.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:15 smithi062 ceph-mon[29843]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:15.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:15 smithi146 ceph-mon[25945]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:16.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:40:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:40:17.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:17 smithi062 ceph-mon[25095]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:17.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:17 smithi062 ceph-mon[29843]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:17.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:17 smithi146 ceph-mon[25945]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:19.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:19 smithi062 ceph-mon[25095]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:19.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:19 smithi062 ceph-mon[29843]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:19.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:19 smithi146 ceph-mon[25945]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:21.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:40:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:20] "GET /metrics HTTP/1.1" 200 214923 "" "Prometheus/2.33.4" 2024-06-01T02:40:21.507 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:21 smithi146 ceph-mon[25945]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:21.508 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:21 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:21.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:21 smithi062 ceph-mon[25095]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:21.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:21 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:21.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:21 smithi062 ceph-mon[29843]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:21.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:21 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:23.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:23 smithi062 ceph-mon[29843]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:23.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:23 smithi062 ceph-mon[25095]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:23.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:23 smithi146 ceph-mon[25945]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:25.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:25 smithi062 ceph-mon[29843]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:25 smithi062 ceph-mon[25095]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:25.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:25 smithi146 ceph-mon[25945]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:26.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:40:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:40:27.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:27 smithi062 ceph-mon[25095]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:27.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:27 smithi062 ceph-mon[29843]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:27.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:27 smithi146 ceph-mon[25945]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:29.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:29 smithi062 ceph-mon[25095]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:29.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:29 smithi062 ceph-mon[29843]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:29.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:29 smithi146 ceph-mon[25945]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:31.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:40:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:30] "GET /metrics HTTP/1.1" 200 214923 "" "Prometheus/2.33.4" 2024-06-01T02:40:31.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:31 smithi146 ceph-mon[25945]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:31.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:31 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:31.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:31 smithi062 ceph-mon[25095]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:31.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:31 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:31.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:31 smithi062 ceph-mon[29843]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:31.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:31 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:33.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:33 smithi146 ceph-mon[25945]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:33.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:33 smithi062 ceph-mon[25095]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:33.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:33 smithi062 ceph-mon[29843]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:34.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:40:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:40:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:40:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:40:34.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:40:34.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:40:34.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:40:34.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:40:34.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:40:34.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:40:34.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:40:34.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:40:35.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:35 smithi146 ceph-mon[25945]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:35.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:35 smithi062 ceph-mon[25095]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:35.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:35 smithi062 ceph-mon[29843]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:36.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:40:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:40:37.564 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:37 smithi062 ceph-mon[25095]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:37.565 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:37 smithi062 ceph-mon[29843]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:37.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:37 smithi146 ceph-mon[25945]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:39.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:39 smithi146 ceph-mon[25945]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:39.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:39 smithi062 ceph-mon[25095]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:39.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:39 smithi062 ceph-mon[29843]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:41.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:40:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:40] "GET /metrics HTTP/1.1" 200 214874 "" "Prometheus/2.33.4" 2024-06-01T02:40:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:41 smithi146 ceph-mon[25945]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:41 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:41.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:41 smithi062 ceph-mon[25095]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:41.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:41 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:41.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:41 smithi062 ceph-mon[29843]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:41.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:41 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:43.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:43 smithi146 ceph-mon[25945]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:43.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:43 smithi062 ceph-mon[25095]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:43.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:43 smithi062 ceph-mon[29843]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:45 smithi146 ceph-mon[25945]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:45.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:45 smithi062 ceph-mon[25095]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:45.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:45 smithi062 ceph-mon[29843]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:46.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:40:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:40:47.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:47 smithi146 ceph-mon[25945]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:47.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:47 smithi062 ceph-mon[25095]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:47.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:47 smithi062 ceph-mon[29843]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:48 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:40:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:48 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:40:48.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:48 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:40:48.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:48 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:40:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:48 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:40:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:48 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:40:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:48 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:40:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:48 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:40:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:48 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:40:49.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:49 smithi146 ceph-mon[25945]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:49.771 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:49 smithi062 ceph-mon[25095]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:49.771 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:49 smithi062 ceph-mon[29843]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:51.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:40:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:50] "GET /metrics HTTP/1.1" 200 214891 "" "Prometheus/2.33.4" 2024-06-01T02:40:51.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:51 smithi146 ceph-mon[25945]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:51.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:51 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:51.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:51 smithi062 ceph-mon[25095]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:51.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:51 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:51 smithi062 ceph-mon[29843]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:51 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:40:53.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:53 smithi146 ceph-mon[25945]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:53.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:53 smithi062 ceph-mon[25095]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:53.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:53 smithi062 ceph-mon[29843]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:55.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:55 smithi146 ceph-mon[25945]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:55.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:55 smithi062 ceph-mon[25095]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:55.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:55 smithi062 ceph-mon[29843]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:56.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:40:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:40:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:40:57.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:57 smithi146 ceph-mon[25945]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:57.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:57 smithi062 ceph-mon[25095]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:57.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:57 smithi062 ceph-mon[29843]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:40:59.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:40:59 smithi146 ceph-mon[25945]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:59.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:40:59 smithi062 ceph-mon[25095]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:40:59.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:40:59 smithi062 ceph-mon[29843]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:01.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:41:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:00] "GET /metrics HTTP/1.1" 200 214891 "" "Prometheus/2.33.4" 2024-06-01T02:41:01.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:01 smithi146 ceph-mon[25945]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:01.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:01 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:01.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:01 smithi062 ceph-mon[25095]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:01.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:01 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:01.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:01 smithi062 ceph-mon[29843]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:01.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:01 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:03.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:03 smithi146 ceph-mon[25945]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:03.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:03 smithi062 ceph-mon[25095]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:03.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:03 smithi062 ceph-mon[29843]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:05.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:05 smithi146 ceph-mon[25945]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:05.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:05 smithi062 ceph-mon[25095]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:05.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:05 smithi062 ceph-mon[29843]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:06.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:41:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:41:07.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:07 smithi146 ceph-mon[25945]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:07.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:07 smithi062 ceph-mon[25095]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:07.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:07 smithi062 ceph-mon[29843]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:09.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:09 smithi146 ceph-mon[25945]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:09.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:09 smithi062 ceph-mon[25095]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:09.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:09 smithi062 ceph-mon[29843]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:11.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:41:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:10] "GET /metrics HTTP/1.1" 200 214891 "" "Prometheus/2.33.4" 2024-06-01T02:41:11.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:11 smithi146 ceph-mon[25945]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:11.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:11 smithi062 ceph-mon[25095]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:11.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:11 smithi062 ceph-mon[29843]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:12.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:12 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:12.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:12 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:12.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:12 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:13.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:13 smithi146 ceph-mon[25945]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:13.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:13 smithi062 ceph-mon[25095]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:13.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:13 smithi062 ceph-mon[29843]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:15.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:15 smithi146 ceph-mon[25945]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:15.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:15 smithi062 ceph-mon[25095]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:15.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:15 smithi062 ceph-mon[29843]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:16.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:41:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:41:17.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:17 smithi146 ceph-mon[25945]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:17.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:17 smithi062 ceph-mon[25095]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:17.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:17 smithi062 ceph-mon[29843]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:19.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:19 smithi146 ceph-mon[25945]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:19.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:19 smithi062 ceph-mon[25095]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:19.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:19 smithi062 ceph-mon[29843]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:41:21.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:41:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:20] "GET /metrics HTTP/1.1" 200 214899 "" "Prometheus/2.33.4" 2024-06-01T02:41:21.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:21 smithi146 ceph-mon[25945]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:21.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:21 smithi062 ceph-mon[25095]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:21.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:21 smithi062 ceph-mon[29843]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:22.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:22 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:22.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:22 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:22.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:22 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:23.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:23 smithi146 ceph-mon[25945]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:23.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:23 smithi062 ceph-mon[25095]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:23.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:23 smithi062 ceph-mon[29843]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:25.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:25 smithi146 ceph-mon[25945]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:25.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:25 smithi062 ceph-mon[25095]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:25.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:25 smithi062 ceph-mon[29843]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:26.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:41:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:41:27.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:27 smithi146 ceph-mon[25945]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:27.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:27 smithi062 ceph-mon[29843]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:27.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:27 smithi062 ceph-mon[25095]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:29.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:29 smithi146 ceph-mon[25945]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:29.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:29 smithi062 ceph-mon[25095]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:29.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:29 smithi062 ceph-mon[29843]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:30.907 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:41:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:30] "GET /metrics HTTP/1.1" 200 214899 "" "Prometheus/2.33.4" 2024-06-01T02:41:31.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:31 smithi062 ceph-mon[25095]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:31.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:31 smithi062 ceph-mon[29843]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:31.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:31 smithi146 ceph-mon[25945]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:31.738 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-06-01T02:41:32.698 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:32 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:32.699 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:32 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:32.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:32 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:33.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:33 smithi062 ceph-mon[25095]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:33 smithi062 ceph-mon[29843]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:33.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:33 smithi146 ceph-mon[25945]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:34.286 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-06-01T02:41:34.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:41:34.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:41:34.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:41:34.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:41:34.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:41:34.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:41:34.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:41:34.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:41:34.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:41:34.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:41:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:41:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:41:35.502 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:35 smithi062 ceph-mon[25095]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:35.502 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:35 smithi062 ceph-mon[29843]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:35.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:35 smithi146 ceph-mon[25945]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:36.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:41:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:41:36.711 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph config set global log_to_journald false --force' 2024-06-01T02:41:37.676 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:37 smithi062 ceph-mon[25095]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:37.677 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:37 smithi062 ceph-mon[29843]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:37.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:37 smithi146 ceph-mon[25945]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:39.201 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-06-01T02:41:39.695 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:39 smithi062 ceph-mon[29843]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:39.695 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:39 smithi062 ceph-mon[25095]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:39.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:39 smithi146 ceph-mon[25945]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:40.726 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:41:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:40] "GET /metrics HTTP/1.1" 200 214890 "" "Prometheus/2.33.4" 2024-06-01T02:41:41.043 INFO:teuthology.orchestra.run.smithi062.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:41.656 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[25095]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:41.656 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:41:41.656 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:41:41.656 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:41:41.656 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:41:41.657 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[29843]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:41.657 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:41:41.657 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:41:41.657 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:41:41.657 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:41 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:41:41.679 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-01T02:41:41.691 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-06-01T02:41:41.692 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-06-01T02:41:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:41 smithi146 ceph-mon[25945]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:41:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:41 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:41:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:41 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:41:41.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:41 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:41:42.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:42 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:42.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:42 smithi146 ceph-mon[25945]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:42.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:42 smithi146 ceph-mon[25945]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:42.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:42 smithi146 ceph-mon[25945]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:42.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:42.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[25095]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:42.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[25095]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:42.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[25095]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:42.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:42.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[29843]: from='client.24892 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:42.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[29843]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:42.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:42 smithi062 ceph-mon[29843]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:41:43.620 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:43 smithi062 ceph-mon[25095]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:43.620 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:43 smithi062 ceph-mon[29843]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:43.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:43 smithi146 ceph-mon[25945]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:44.023 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (2m) 117s ago 3m 19.4M - ba2b418f427c 7ecddd10ccf3 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (2m) 117s ago 2m 46.2M - 8.3.5 dad864ee21e9 587510bef419 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (2m) 117s ago 2m 70.8M - 3.5 e1d6a67b021e c9e47aa4b33e 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443 running (7m) 117s ago 7m 411M - 17.2.0 e1d6a67b021e 528940eea75a 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:9283 running (9m) 117s ago 9m 456M - 17.2.0 e1d6a67b021e 222c6c0c8675 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (9m) 117s ago 9m 46.6M 2048M 17.2.0 e1d6a67b021e 74c10effbe9a 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (7m) 117s ago 7m 39.3M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (8m) 117s ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (3m) 117s ago 3m 16.0M - 1dbe0e931976 b7a20b957048 2024-06-01T02:41:44.892 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (3m) 117s ago 3m 14.1M - 1dbe0e931976 bcacd1daad1e 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (7m) 117s ago 7m 49.8M 4016M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (6m) 117s ago 6m 51.4M 4016M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (6m) 117s ago 6m 49.1M 4016M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (6m) 117s ago 6m 49.1M 4016M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (5m) 117s ago 5m 51.9M 4272M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (5m) 117s ago 5m 48.6M 4272M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (4m) 117s ago 4m 49.1M 4272M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (4m) 117s ago 4m 51.1M 4272M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (2m) 117s ago 3m 44.4M - 514e6a882f6e ca0af62bf736 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (2m) 117s ago 2m 72.5M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (2m) 117s ago 2m 71.2M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (2m) 117s ago 2m 72.1M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:41:44.893 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (2m) 117s ago 2m 70.2M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:41:45.416 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:41:45.416 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:41:45.416 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-01T02:41:45.416 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": {}, 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-06-01T02:41:45.417 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:41:45.418 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:41:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:45 smithi146 ceph-mon[25945]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:45.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:45 smithi146 ceph-mon[25945]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:45.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:45 smithi146 ceph-mon[25945]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:45.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:45 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1416260020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:41:45.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[25095]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:45.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[25095]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[25095]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1416260020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:41:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[29843]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[29843]: from='client.24898 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[29843]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:45 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1416260020' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "", 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba image" 2024-06-01T02:41:45.857 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:41:46.301 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:41:46.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:41:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:46] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:41:46.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:46 smithi146 ceph-mon[25945]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:46.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:46 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3635407847' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:41:46.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:46 smithi062 ceph-mon[25095]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:46.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:46 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3635407847' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:41:46.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:46 smithi062 ceph-mon[29843]: from='client.15015 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:46.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:46 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3635407847' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:41:47.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:47 smithi146 ceph-mon[25945]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:47.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:47 smithi146 ceph-mon[25945]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:47.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:47 smithi062 ceph-mon[25095]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:47.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:47 smithi062 ceph-mon[25095]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:47.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:47 smithi062 ceph-mon[29843]: from='client.15027 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:41:47.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:47 smithi062 ceph-mon[29843]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:49.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:49 smithi062 ceph-mon[25095]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:49.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:49 smithi062 ceph-mon[29843]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:49.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:49 smithi146 ceph-mon[25945]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:51.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:41:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:50] "GET /metrics HTTP/1.1" 200 214890 "" "Prometheus/2.33.4" 2024-06-01T02:41:51.758 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:51 smithi146 ceph-mon[25945]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:51.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:51 smithi062 ceph-mon[25095]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:51.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:51 smithi062 ceph-mon[29843]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:52.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:52 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:52.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:52 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:52.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:52 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:41:53.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:53 smithi062 ceph-mon[25095]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:53.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:53 smithi062 ceph-mon[29843]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:53.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:53 smithi146 ceph-mon[25945]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:55.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:55 smithi062 ceph-mon[25095]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:55.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:55 smithi062 ceph-mon[29843]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:55 smithi146 ceph-mon[25945]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:56.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:41:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:41:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:41:57.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:57 smithi062 ceph-mon[25095]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:57.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:57 smithi062 ceph-mon[29843]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:57.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:57 smithi146 ceph-mon[25945]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:41:59.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:41:59 smithi062 ceph-mon[25095]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:59.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:41:59 smithi062 ceph-mon[29843]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:41:59.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:41:59 smithi146 ceph-mon[25945]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:01.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:42:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:00] "GET /metrics HTTP/1.1" 200 214890 "" "Prometheus/2.33.4" 2024-06-01T02:42:01.762 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:01 smithi062 ceph-mon[29843]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:01.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:01 smithi062 ceph-mon[25095]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:01.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:01 smithi146 ceph-mon[25945]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:02.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:02 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:02.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:02 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:02.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:02 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:03.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:03 smithi062 ceph-mon[25095]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:03.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:03 smithi062 ceph-mon[29843]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:03.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:03 smithi146 ceph-mon[25945]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:05.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:05 smithi062 ceph-mon[25095]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:05.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:05 smithi062 ceph-mon[29843]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:05.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:05 smithi146 ceph-mon[25945]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:06.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:06] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:42:07.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:07 smithi062 ceph-mon[25095]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:07.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:07 smithi062 ceph-mon[29843]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:07.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:07 smithi146 ceph-mon[25945]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:09.715 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:09 smithi062 ceph-mon[25095]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:09.716 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:09 smithi062 ceph-mon[29843]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:09.780 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:09 smithi146 ceph-mon[25945]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:11.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[25095]: Upgrade: Target is version 19.0.0-2946-g3d89accf (unknown) 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[25095]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, digests ['quay.ceph.io/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9'] 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[25095]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:10] "GET /metrics HTTP/1.1" 200 214846 "" "Prometheus/2.33.4" 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:11.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[29843]: Upgrade: Target is version 19.0.0-2946-g3d89accf (unknown) 2024-06-01T02:42:11.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[29843]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, digests ['quay.ceph.io/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9'] 2024-06-01T02:42:11.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:42:11.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:11.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:10 smithi062 ceph-mon[29843]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-01T02:42:11.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:10 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:11.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:10 smithi146 ceph-mon[25945]: Upgrade: Target is version 19.0.0-2946-g3d89accf (unknown) 2024-06-01T02:42:11.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:10 smithi146 ceph-mon[25945]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, digests ['quay.ceph.io/ceph-ci/ceph@sha256:fe72ccfadf2b7c7c133c6379dcb4e96c4db2f7f512b19bd8eb3efb31f1d8c5d9'] 2024-06-01T02:42:11.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:10 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:42:11.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:10 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:11.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:10 smithi146 ceph-mon[25945]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-01T02:42:12.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:11 smithi062 ceph-mon[25095]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:12.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:11 smithi062 ceph-mon[25095]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba on smithi146 2024-06-01T02:42:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:11 smithi062 ceph-mon[29843]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:11 smithi062 ceph-mon[29843]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba on smithi146 2024-06-01T02:42:12.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:11 smithi146 ceph-mon[25945]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:12.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:11 smithi146 ceph-mon[25945]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba on smithi146 2024-06-01T02:42:13.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:12 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:12 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:13.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:12 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:13.760 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:13 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:13.760 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:13 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:13.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:14.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:13 smithi062 ceph-mon[25095]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:14.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:13 smithi062 ceph-mon[29843]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:14.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:13 smithi146 ceph-mon[25945]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:16.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:15 smithi062 ceph-mon[25095]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:16.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:15 smithi062 ceph-mon[29843]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:16.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:15 smithi146 ceph-mon[25945]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:16.640 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:42:16.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:16] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (3m) 2m ago 3m 19.4M - ba2b418f427c 7ecddd10ccf3 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (3m) 2m ago 3m 46.2M - 8.3.5 dad864ee21e9 587510bef419 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (2m) 2m ago 2m 70.8M - 3.5 e1d6a67b021e c9e47aa4b33e 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443 running (8m) 2m ago 8m 411M - 17.2.0 e1d6a67b021e 528940eea75a 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:9283 running (9m) 2m ago 9m 456M - 17.2.0 e1d6a67b021e 222c6c0c8675 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (9m) 2m ago 10m 46.6M 2048M 17.2.0 e1d6a67b021e 74c10effbe9a 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (8m) 2m ago 8m 39.3M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:42:17.252 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (8m) 2m ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (4m) 2m ago 4m 16.0M - 1dbe0e931976 b7a20b957048 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (4m) 2m ago 4m 14.1M - 1dbe0e931976 bcacd1daad1e 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (7m) 2m ago 7m 49.8M 4016M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (7m) 2m ago 7m 51.4M 4016M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (7m) 2m ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (6m) 2m ago 6m 49.1M 4016M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (6m) 2m ago 6m 51.9M 4272M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (5m) 2m ago 5m 48.6M 4272M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (5m) 2m ago 5m 49.1M 4272M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (5m) 2m ago 5m 51.1M 4272M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (3m) 2m ago 4m 44.4M - 514e6a882f6e ca0af62bf736 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (2m) 2m ago 2m 72.5M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (2m) 2m ago 2m 71.2M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (2m) 2m ago 2m 72.1M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:42:17.253 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (2m) 2m ago 2m 70.2M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:42:17.618 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:42:17.618 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": {}, 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:42:17.619 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-06-01T02:42:17.620 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:42:17.620 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "0/23 daemons upgraded", 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba image on host smithi146" 2024-06-01T02:42:17.931 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:42:17.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:17 smithi062 ceph-mon[25095]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:17.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:17 smithi062 ceph-mon[25095]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:17.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:17 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3495668231' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:42:17.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:17 smithi062 ceph-mon[29843]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:17.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:17 smithi062 ceph-mon[29843]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:17.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:17 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3495668231' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:42:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:17 smithi146 ceph-mon[25945]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:17 smithi146 ceph-mon[25945]: from='client.15039 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:18.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:17 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3495668231' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:42:18.314 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:42:19.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:18 smithi146 ceph-mon[25945]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:19.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:18 smithi146 ceph-mon[25945]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:19.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:18 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3083684652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:42:19.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:18 smithi062 ceph-mon[25095]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:19.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:18 smithi062 ceph-mon[25095]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:18 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3083684652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:42:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:18 smithi062 ceph-mon[29843]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:18 smithi062 ceph-mon[29843]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:18 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3083684652' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:42:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:19 smithi146 ceph-mon[25945]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:20.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:19 smithi146 ceph-mon[25945]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:20.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:19 smithi062 ceph-mon[25095]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:19 smithi062 ceph-mon[25095]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:19 smithi062 ceph-mon[29843]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:19 smithi062 ceph-mon[29843]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:20.899 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:42:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:20] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-06-01T02:42:22.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:21 smithi146 ceph-mon[25945]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:22.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:21 smithi062 ceph-mon[25095]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:21 smithi062 ceph-mon[29843]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:23.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:22 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:23.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:22 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:23.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:22 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:23.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:42:23.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:23.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:23.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:24.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:23 smithi146 ceph-mon[25945]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:24.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:23 smithi062 ceph-mon[25095]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:24.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:23 smithi062 ceph-mon[29843]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:26.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:25 smithi146 ceph-mon[25945]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:26.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:25 smithi062 ceph-mon[25095]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:26.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:25 smithi062 ceph-mon[29843]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:26.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:26] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:42:28.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:27 smithi146 ceph-mon[25945]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:28.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:27 smithi062 ceph-mon[25095]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:28.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:27 smithi062 ceph-mon[29843]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:30.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:29 smithi146 ceph-mon[25945]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:30.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:29 smithi062 ceph-mon[25095]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:30.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:29 smithi062 ceph-mon[29843]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:30.909 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:42:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:30] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-06-01T02:42:32.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:31 smithi146 ceph-mon[25945]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:32.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:31 smithi062 ceph-mon[25095]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:32.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:31 smithi062 ceph-mon[29843]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:33.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:32 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:33.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:32 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:32 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:33.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:33 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:42:33.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:33.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:33 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:33.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:33 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:33.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:33 smithi146 ceph-mon[25945]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:33 smithi062 ceph-mon[25095]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:34.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:33 smithi062 ceph-mon[29843]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:35.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:42:35.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:34 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:42:35.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:42:35.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:34 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:42:35.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:42:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:34 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:42:36.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:35 smithi146 ceph-mon[25945]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:36.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:35 smithi062 ceph-mon[25095]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:35 smithi062 ceph-mon[29843]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:36.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[27120]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:36] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:42:38.065 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:37 smithi062 ceph-mon[25095]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:38.065 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:37 smithi062 ceph-mon[29843]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:38.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:37 smithi146 ceph-mon[25945]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:40.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:39 smithi146 ceph-mon[25945]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:39 smithi062 ceph-mon[25095]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:40.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:39 smithi062 ceph-mon[29843]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:40.912 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:42:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:40] "GET /metrics HTTP/1.1" 200 214835 "" "Prometheus/2.33.4" 2024-06-01T02:42:42.040 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:41 smithi146 ceph-mon[25945]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:42.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:41 smithi062 ceph-mon[25095]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:41 smithi062 ceph-mon[29843]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:43.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:42 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:43.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:42 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:43.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:42 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:43.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:42:43.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:43.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:43.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:43.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:43 smithi146 ceph-mon[25945]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:43 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:43 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:42:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:43 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:42:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:43 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:42:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:43 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:42:44.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[25095]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[29843]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:42:44.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:42:44.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:43 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:42:44.621 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:44 smithi146 systemd[1]: Stopping Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:42:44.622 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:44 smithi146 bash[46822]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr.x" found: no such container 2024-06-01T02:42:44.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:44 smithi146 ceph-mon[25945]: Upgrade: Updating mgr.x 2024-06-01T02:42:44.929 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:44 smithi146 ceph-mon[25945]: Deploying daemon mgr.x on smithi146 2024-06-01T02:42:44.929 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:44 smithi146 podman[46834]: 2024-06-01 02:42:44.622136995 +0000 UTC m=+0.130889464 container died 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD) 2024-06-01T02:42:45.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 podman[46834]: 2024-06-01 02:42:45.001311174 +0000 UTC m=+0.510063633 container cleanup 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, RELEASE=HEAD, name=centos-stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vcs-type=git, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, architecture=x86_64) 2024-06-01T02:42:45.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 bash[46834]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x 2024-06-01T02:42:45.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 bash[46861]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr.x" found: no such container 2024-06-01T02:42:45.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:44 smithi062 ceph-mon[25095]: Upgrade: Updating mgr.x 2024-06-01T02:42:45.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:44 smithi062 ceph-mon[25095]: Deploying daemon mgr.x on smithi146 2024-06-01T02:42:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:44 smithi062 ceph-mon[29843]: Upgrade: Updating mgr.x 2024-06-01T02:42:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:44 smithi062 ceph-mon[29843]: Deploying daemon mgr.x on smithi146 2024-06-01T02:42:45.669 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 podman[46849]: 2024-06-01 02:42:45.252495599 +0000 UTC m=+0.621758676 container remove 528940eea75abcd4b1e548915b25fa274c1cdd4d1627fe1c4bd3803982097f8d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, version=8, distribution-scope=public, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, RELEASE=HEAD) 2024-06-01T02:42:45.669 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:42:45.669 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-06-01T02:42:45.669 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 systemd[1]: Stopped Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:42:45.670 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service: Consumed 16.769s CPU time. 2024-06-01T02:42:45.670 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 systemd[1]: Starting Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:42:45.941 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:45 smithi146 ceph-mon[25945]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:45.941 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 podman[46959]: 2024-06-01 02:42:45.572828582 +0000 UTC m=+0.017560190 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:42:45.942 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 podman[46959]: 2024-06-01 02:42:45.689350428 +0000 UTC m=+0.134082023 container create 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-01T02:42:45.942 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 podman[46959]: 2024-06-01 02:42:45.914598193 +0000 UTC m=+0.359329796 container init 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:42:45.942 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 podman[46959]: 2024-06-01 02:42:45.917584697 +0000 UTC m=+0.362316301 container start 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-01T02:42:45.942 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 ceph-mgr[46978]: -- 172.21.15.146:0/1414379249 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d0942e5a0 con 0x560d09403000 2024-06-01T02:42:46.097 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:45 smithi062 ceph-mon[25095]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:46.097 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:46 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:46.059Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=4 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:42:46.098 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:45 smithi062 ceph-mon[29843]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:46.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 bash[46959]: 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb 2024-06-01T02:42:46.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:45 smithi146 systemd[1]: Started Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:42:46.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:46.078+0000 7f3b74676200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:42:46.620 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:46.279+0000 7f3b74676200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:42:46.620 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:46.399+0000 7f3b74676200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:42:46.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:46.723+0000 7f3b74676200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:42:46.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:46.814+0000 7f3b74676200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:42:47.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:47 smithi146 ceph-mon[25945]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:47.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:47 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:47.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:47 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:42:47.390 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:47 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[29843]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[25095]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:42:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:47 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:42:48.392 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:48.106+0000 7f3b74676200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:42:48.392 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:48.308+0000 7f3b74676200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:42:48.654 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:42:48.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:48.485+0000 7f3b74676200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:42:49.038 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:48.782+0000 7f3b74676200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:42:49.039 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:48.875+0000 7f3b74676200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:42:49.273 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:42:49.273 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (3m) 3m ago 4m 19.4M - ba2b418f427c 7ecddd10ccf3 2024-06-01T02:42:49.273 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (3m) 3m ago 3m 46.2M - 8.3.5 dad864ee21e9 587510bef419 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (3m) 3m ago 3m 70.8M - 3.5 e1d6a67b021e c9e47aa4b33e 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283 starting - - - - 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 222c6c0c8675 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (10m) 3m ago 10m 46.6M 2048M 17.2.0 e1d6a67b021e 74c10effbe9a 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (8m) 3m ago 8m 39.3M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (9m) 3m ago 9m 40.4M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (4m) 3m ago 4m 16.0M - 1dbe0e931976 b7a20b957048 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (4m) 3m ago 4m 14.1M - 1dbe0e931976 bcacd1daad1e 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (8m) 3m ago 8m 49.8M 4016M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8m) 3m ago 8m 51.4M 4016M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (7m) 3m ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (7m) 3m ago 7m 49.1M 4016M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:42:49.274 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (6m) 3m ago 6m 51.9M 4272M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (6m) 3m ago 6m 48.6M 4272M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (6m) 3m ago 6m 49.1M 4272M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (5m) 3m ago 5m 51.1M 4272M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (3m) 3m ago 4m 44.4M - 514e6a882f6e ca0af62bf736 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (3m) 3m ago 3m 72.5M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (3m) 3m ago 3m 71.2M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (3m) 3m ago 3m 72.1M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:42:49.275 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (3m) 3m ago 3m 70.2M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:42:49.292 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:49 smithi146 ceph-mon[25945]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:49.292 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:49 smithi146 ceph-mon[25945]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:49.292 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:49 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:49.093+0000 7f3b74676200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:42:49.357 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:49 smithi062 ceph-mon[25095]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:49.357 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:49 smithi062 ceph-mon[25095]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:49.357 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:49 smithi062 ceph-mon[29843]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:49.357 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:49 smithi062 ceph-mon[29843]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:49.645 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": {}, 2024-06-01T02:42:49.646 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:42:49.647 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:42:49.647 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:42:49.647 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:42:49.647 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-06-01T02:42:49.647 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:42:49.647 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:42:49.653 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:49 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:49.292+0000 7f3b74676200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:42:49.954 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:42:49.955 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:42:49.955 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:42:49.955 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-06-01T02:42:49.955 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "", 2024-06-01T02:42:49.955 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mgr daemons" 2024-06-01T02:42:49.955 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:42:50.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:49 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:49.947+0000 7f3b74676200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:42:50.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:50 smithi146 ceph-mon[25945]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:50.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:50 smithi146 ceph-mon[25945]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:50.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:50 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1953467303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:42:50.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-mon[29843]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:50.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-mon[29843]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:50.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1953467303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:42:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-mon[25095]: from='client.24982 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-mon[25095]: from='client.15093 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1953467303' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:42:50.320 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:42:50.674 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:50.389+0000 7f3b74676200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:42:50.674 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:50.496+0000 7f3b74676200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:42:50.915 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:42:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:50] "GET /metrics HTTP/1.1" 200 214824 "" "Prometheus/2.33.4" 2024-06-01T02:42:50.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:50 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:50.706+0000 7f3b74676200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[29843]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[29843]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3508866664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[25095]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[25095]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3508866664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:42:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:51 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:51.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-mon[25945]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:51.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-mon[25945]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:42:51.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3508866664' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:42:51.299 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:51.578 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:51.299+0000 7f3b74676200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:42:51.578 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:51.400+0000 7f3b74676200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:42:51.916 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:51.664+0000 7f3b74676200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:42:51.916 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:51.807+0000 7f3b74676200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:42:51.916 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:51 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:51.916+0000 7f3b74676200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:42:52.195 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:52 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:52.036+0000 7f3b74676200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:42:52.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:52 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:52.357+0000 7f3b74676200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:42:52.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:52 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:52.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:52 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:53.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:52 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:53.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:52 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:52 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:42:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:52 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:42:53.216 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:52 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:52.984+0000 7f3b74676200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:42:53.216 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:53.215+0000 7f3b74676200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:42:53.528 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:42:53] ENGINE Bus STARTING 2024-06-01T02:42:53.528 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: CherryPy Checker: 2024-06-01T02:42:53.529 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: The Application mounted at '' has an empty config. 2024-06-01T02:42:53.529 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:42:53.529 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:42:53] ENGINE Serving on http://:::9283 2024-06-01T02:42:53.529 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:42:53] ENGINE Bus STARTED 2024-06-01T02:42:53.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: Standby manager daemon x restarted 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: Standby manager daemon x started 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: Standby manager daemon x restarted 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: Standby manager daemon x started 2024-06-01T02:42:53.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:42:53.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:42:53.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:42:53.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:42:53.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:53.475Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=7 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:53.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:42:53.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:53.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:42:53.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:42:53 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:42:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:42:53.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: Standby manager daemon x restarted 2024-06-01T02:42:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: Standby manager daemon x started 2024-06-01T02:42:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:42:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:42:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:42:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:53 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:42:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:54 smithi146 ceph-mon[25945]: mgrmap e22: y(active, since 5m), standbys: x 2024-06-01T02:42:55.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:54 smithi062 ceph-mon[25095]: mgrmap e22: y(active, since 5m), standbys: x 2024-06-01T02:42:55.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:54 smithi062 ceph-mon[29843]: mgrmap e22: y(active, since 5m), standbys: x 2024-06-01T02:42:55.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:55 smithi146 ceph-mon[25945]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:56.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:55 smithi062 ceph-mon[25095]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:56.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:55 smithi062 ceph-mon[29843]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:56.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:42:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:42:56] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:42:57.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:57 smithi146 ceph-mon[25945]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:58.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:57 smithi062 ceph-mon[25095]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:58.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:57 smithi062 ceph-mon[29843]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:42:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:59 smithi146 ceph-mon[25945]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:42:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:42:59 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:43:00.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:59 smithi062 ceph-mon[25095]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:42:59 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:43:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:59 smithi062 ceph-mon[29843]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:00.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:42:59 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:43:00.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:43:00.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:00.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-01T02:43:00.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: Failing over to other MGR 2024-06-01T02:43:00.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-01T02:43:00.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-mon[25945]: osdmap e97: 8 total, 8 up, 8 in 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:00] ENGINE Bus STOPPING 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:00] ENGINE Bus STOPPED 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:00] ENGINE Bus STARTING 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:00] ENGINE Serving on http://:::9283 2024-06-01T02:43:00.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:00] ENGINE Bus STARTED 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: Failing over to other MGR 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-01T02:43:00.954 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[29843]: osdmap e97: 8 total, 8 up, 8 in 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: Upgrade: Need to upgrade myself (mgr.y) 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: Failing over to other MGR 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: from='mgr.24331 172.21.15.62:0/1373659688' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd=[{"prefix": "mgr fail", "who": "y"}]: dispatch 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mon[25095]: osdmap e97: 8 total, 8 up, 8 in 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:00.640+0000 7f76367fd700 -1 mgr handle_mgr_map I was active but no longer am 2024-06-01T02:43:00.955 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setuser ceph since I am not root 2024-06-01T02:43:00.956 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ignoring --setgroup ceph since I am not root 2024-06-01T02:43:00.956 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-mgr[25324]: -- 172.21.15.62:0/3814149614 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be11358340 con 0x55be11331000 2024-06-01T02:43:00.956 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:00.818+0000 7ff032f28000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:43:00.956 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:00 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:00.954+0000 7ff032f28000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:43:01.262 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:01.056+0000 7ff032f28000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:43:01.638 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:01.385+0000 7ff032f28000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:43:01.638 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:01.472+0000 7ff032f28000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: mgrmap e23: x(active, starting, since 0.996867s) 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:43:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: Manager daemon x is now available 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: Queued rgw.foo for migration 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: Queued rgw.smpl for migration 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-06-01T02:43:01.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:01.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:01 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: mgrmap e23: x(active, starting, since 0.996867s) 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:43:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: Manager daemon x is now available 2024-06-01T02:43:02.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: Queued rgw.foo for migration 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: Queued rgw.smpl for migration 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24331 ' entity='mgr.y' cmd='[{"prefix": "mgr fail", "who": "y"}]': finished 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: mgrmap e23: x(active, starting, since 0.996867s) 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:43:02.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: Manager daemon x is now available 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: Queued rgw.foo for migration 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: Queued rgw.smpl for migration 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'foo', 'service_name': 'rgw.foo', 'service_type': 'rgw', 'spec': {'rgw_frontend_port': 8000, 'rgw_realm': 'r', 'rgw_zone': 'z'}} 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: No Migration is needed for rgw spec: {'placement': {'count': 2}, 'service_id': 'smpl', 'service_name': 'rgw.smpl', 'service_type': 'rgw'} 2024-06-01T02:43:02.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.018 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:01 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:02.924 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[25095]: Deploying cephadm binary to smithi146 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[25095]: Deploying cephadm binary to smithi062 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[25095]: mgrmap e24: x(active, since 2s) 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[25095]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:02.664+0000 7ff032f28000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:02.838+0000 7ff032f28000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:43:02.925 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[29843]: Deploying cephadm binary to smithi146 2024-06-01T02:43:02.926 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[29843]: Deploying cephadm binary to smithi062 2024-06-01T02:43:02.926 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[29843]: mgrmap e24: x(active, since 2s) 2024-06-01T02:43:02.926 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:02.926 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-mon[29843]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:02.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:02 smithi146 ceph-mon[25945]: Deploying cephadm binary to smithi146 2024-06-01T02:43:02.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:02 smithi146 ceph-mon[25945]: Deploying cephadm binary to smithi062 2024-06-01T02:43:02.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:02 smithi146 ceph-mon[25945]: mgrmap e24: x(active, since 2s) 2024-06-01T02:43:02.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:02 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:02.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:02 smithi146 ceph-mon[25945]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:03.254 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:02 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:02.924+0000 7ff032f28000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:43:03.254 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:03.254+0000 7ff032f28000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:43:03.510 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:03.350+0000 7ff032f28000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:43:03.510 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:03.447+0000 7ff032f28000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:43:03.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:01] ENGINE Bus STARTING 2024-06-01T02:43:03.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:02] ENGINE Serving on https://172.21.15.146:7150 2024-06-01T02:43:03.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:02] ENGINE Serving on http://172.21.15.146:8765 2024-06-01T02:43:03.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:02] ENGINE Bus STARTED 2024-06-01T02:43:03.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[25095]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:03.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:03.703+0000 7ff032f28000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:43:03.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:01] ENGINE Bus STARTING 2024-06-01T02:43:03.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:02] ENGINE Serving on https://172.21.15.146:7150 2024-06-01T02:43:03.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:02] ENGINE Serving on http://172.21.15.146:8765 2024-06-01T02:43:03.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:02] ENGINE Bus STARTED 2024-06-01T02:43:03.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-mon[29843]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:03.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:43:03.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:43:03.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:43:03.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:43:03.764 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:43:03.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:43:03.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:03 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:01] ENGINE Bus STARTING 2024-06-01T02:43:03.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:03 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:02] ENGINE Serving on https://172.21.15.146:7150 2024-06-01T02:43:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:03 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:02] ENGINE Serving on http://172.21.15.146:8765 2024-06-01T02:43:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:03 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:02] ENGINE Bus STARTED 2024-06-01T02:43:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:03 smithi146 ceph-mon[25945]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:04.512 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:04.168+0000 7ff032f28000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:43:04.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:04 smithi146 ceph-mon[25945]: mgrmap e25: x(active, since 4s) 2024-06-01T02:43:05.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:04 smithi062 ceph-mon[25095]: mgrmap e25: x(active, since 4s) 2024-06-01T02:43:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:04 smithi062 ceph-mon[29843]: mgrmap e25: x(active, since 4s) 2024-06-01T02:43:05.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:04.680+0000 7ff032f28000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:43:05.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:04.778+0000 7ff032f28000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:43:05.512 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:05.086+0000 7ff032f28000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:43:05.513 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:05.181+0000 7ff032f28000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:43:05.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:05 smithi146 ceph-mon[25945]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:05.963 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-mon[25095]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:05.963 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-mon[29843]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:05.963 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:05.606+0000 7ff032f28000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:43:05.964 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:05.746+0000 7ff032f28000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:43:05.964 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:05.843+0000 7ff032f28000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:43:05.964 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:05 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:05.963+0000 7ff032f28000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:43:06.262 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:06.060+0000 7ff032f28000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:43:06.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:06] "GET /metrics HTTP/1.1" 200 35152 "" "Prometheus/2.33.4" 2024-06-01T02:43:07.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:06.654+0000 7ff032f28000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:07.054+0000 7ff032f28000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:43:07] ENGINE Bus STARTING 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: CherryPy Checker: 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: The Application mounted at '' has an empty config. 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:43:07] ENGINE Serving on http://:::9283 2024-06-01T02:43:07.322 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: [01/Jun/2024:02:43:07] ENGINE Bus STARTED 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: Standby manager daemon y started 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-01T02:43:07.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:07 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:43:07.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: Standby manager daemon y started 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:07.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: Standby manager daemon y started 2024-06-01T02:43:07.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-01T02:43:07.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:43:07.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-01T02:43:07.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/1974112475' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:43:08.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:43:07.983Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=5 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:43:08.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:08 smithi062 ceph-mon[25095]: mgrmap e26: x(active, since 7s), standbys: y 2024-06-01T02:43:08.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:08 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:43:08.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:08 smithi062 ceph-mon[29843]: mgrmap e26: x(active, since 7s), standbys: y 2024-06-01T02:43:08.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:08 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:43:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:08 smithi146 ceph-mon[25945]: mgrmap e26: x(active, since 7s), standbys: y 2024-06-01T02:43:08.921 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:08 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:43:09.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:09 smithi062 ceph-mon[25095]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-01T02:43:09.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:09 smithi062 ceph-mon[29843]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-01T02:43:09.834 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:09 smithi146 ceph-mon[25945]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-01T02:43:11.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:43:11.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:11 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:11 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:11 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:11 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:11 smithi146 ceph-mon[25945]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-01T02:43:11.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[25095]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:11.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:11 smithi062 ceph-mon[29843]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-01T02:43:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi062 to 2736M 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:12 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi062 to 2736M 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi062 to 2736M 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:12.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:12 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:13.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:13 smithi146 ceph-mon[25945]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:43:13.510 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:13 smithi062 ceph-mon[25095]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:43:13.511 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:13 smithi062 ceph-mon[29843]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:43:13.762 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:13 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=error ts=2024-06-01T02:43:13.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:43:13.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:13 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:43:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.62:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.62 because it doesn't contain any IP SANs" 2024-06-01T02:43:13.763 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:13 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=warn ts=2024-06-01T02:43:13.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.146:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.146 because it doesn't contain any IP SANs" 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:15.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: Adjusting osd_memory_target on smithi146 to 2992M 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:43:15.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:15 smithi146 ceph-mon[25945]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: Adjusting osd_memory_target on smithi146 to 2992M 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[25095]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:43:16.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: Adjusting osd_memory_target on smithi146 to 2992M 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:43:16.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:43:16.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:43:16.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:16.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:43:16.016 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:15 smithi062 ceph-mon[29843]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:43:16.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:16 smithi062 systemd[1]: Stopping Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:16.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:16 smithi062 bash[58479]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter.a" found: no such container 2024-06-01T02:43:16.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:16 smithi062 podman[58490]: 2024-06-01 02:43:16.521977576 +0000 UTC m=+0.121909973 container died b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:16.687 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:16] "GET /metrics HTTP/1.1" 200 38151 "" "Prometheus/2.33.4" 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-01T02:43:16.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:16 smithi146 ceph-mon[25945]: Reconfiguring daemon node-exporter.a on smithi062 2024-06-01T02:43:17.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:43:17.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:43:17.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:43:17.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:43:17.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[25095]: Reconfiguring daemon node-exporter.a on smithi062 2024-06-01T02:43:17.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:17.005 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: Reconfiguring node-exporter.a (dependencies changed)... 2024-06-01T02:43:17.005 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:16 smithi062 ceph-mon[29843]: Reconfiguring daemon node-exporter.a on smithi062 2024-06-01T02:43:17.262 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 podman[58490]: 2024-06-01 02:43:17.003214423 +0000 UTC m=+0.603146798 container cleanup b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:17.262 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 bash[58490]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a 2024-06-01T02:43:17.263 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 bash[58515]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter.a" found: no such container 2024-06-01T02:43:17.686 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 podman[58501]: 2024-06-01 02:43:17.355163697 +0000 UTC m=+0.832460086 container remove b7a20b957048d4639da0fcd5c76bbe18f060bbcffd09b7ca46eada7634a761b4 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:17.686 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:43:17.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-06-01T02:43:17.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 systemd[1]: Stopped Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:17.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service: Consumed 2.700s CPU time. 2024-06-01T02:43:17.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 systemd[1]: Starting Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:17.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 podman[58576]: 2024-06-01 02:43:17.589237537 +0000 UTC m=+0.112026326 container create eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:17.687 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 podman[58576]: 2024-06-01 02:43:17.495812037 +0000 UTC m=+0.018600847 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-01T02:43:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:17 smithi146 ceph-mon[25945]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:18.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-mon[25095]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:18.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-mon[29843]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:18.263 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 podman[58576]: 2024-06-01 02:43:17.77485456 +0000 UTC m=+0.297643340 container init eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:18.263 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 podman[58576]: 2024-06-01 02:43:17.77850869 +0000 UTC m=+0.301297482 container start eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:18.264 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.781Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-01T02:43:18.264 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.781Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-01T02:43:18.264 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.782Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-01T02:43:18.264 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.782Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-01T02:43:18.264 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.782Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-01T02:43:18.264 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=arp 2024-06-01T02:43:18.265 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-01T02:43:18.265 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-01T02:43:18.265 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-01T02:43:18.265 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-01T02:43:18.265 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-01T02:43:18.265 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-01T02:43:18.266 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-01T02:43:18.266 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-01T02:43:18.266 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=edac 2024-06-01T02:43:18.267 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-01T02:43:18.267 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-01T02:43:18.267 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-01T02:43:18.267 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-01T02:43:18.268 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-01T02:43:18.268 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-01T02:43:18.268 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-01T02:43:18.268 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-01T02:43:18.268 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-01T02:43:18.268 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=os 2024-06-01T02:43:18.269 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=stat 2024-06-01T02:43:18.270 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-01T02:43:18.271 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-01T02:43:18.271 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-01T02:43:18.271 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=time 2024-06-01T02:43:18.271 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-01T02:43:18.271 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=uname 2024-06-01T02:43:18.271 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-01T02:43:18.272 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-01T02:43:18.272 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-01T02:43:18.272 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-01T02:43:18.272 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a[58591]: ts=2024-06-01T02:43:17.783Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-01T02:43:18.272 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 bash[58576]: eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 2024-06-01T02:43:18.272 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:43:17 smithi062 systemd[1]: Started Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:19.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:18 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:19.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:18 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:19.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:18 smithi146 ceph-mon[25945]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-01T02:43:19.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:18 smithi146 ceph-mon[25945]: Reconfiguring daemon alertmanager.a on smithi062 2024-06-01T02:43:19.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[25095]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[25095]: Reconfiguring daemon alertmanager.a on smithi062 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[29843]: Reconfiguring alertmanager.a (dependencies changed)... 2024-06-01T02:43:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:18 smithi062 ceph-mon[29843]: Reconfiguring daemon alertmanager.a on smithi062 2024-06-01T02:43:20.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:19 smithi146 ceph-mon[25945]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:19 smithi062 ceph-mon[25095]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:19 smithi062 ceph-mon[29843]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:43:20.702 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:43:20.947 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:20 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[25320]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:43:21.385 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (4m) 11s ago 4m 28.6M - 0.23.0 ba2b418f427c 7ecddd10ccf3 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (4m) 11s ago 4m 55.1M - 8.3.5 dad864ee21e9 587510bef419 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (3m) 11s ago 3m 51.5M - 3.5 e1d6a67b021e c9e47aa4b33e 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283 running (35s) 11s ago 9m 494M - 19.0.0-2946-g3d89accf b045ac545f82 03f23699819f 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:9283 running (11m) 11s ago 11m 400M - 17.2.0 e1d6a67b021e 222c6c0c8675 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (11m) 11s ago 11m 50.6M 2048M 17.2.0 e1d6a67b021e 74c10effbe9a 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (9m) 11s ago 9m 41.7M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (9m) 11s ago 9m 42.2M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 starting - - - - 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (5m) 11s ago 5m 14.5M - 1.3.1 1dbe0e931976 bcacd1daad1e 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (8m) 11s ago 8m 52.3M 2736M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8m) 11s ago 8m 54.2M 2736M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:43:21.386 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (8m) 11s ago 8m 51.0M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (7m) 11s ago 7m 52.3M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (7m) 11s ago 7m 55.3M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (6m) 11s ago 6m 52.1M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (6m) 11s ago 6m 52.2M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (6m) 11s ago 6m 54.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (4m) 11s ago 5m 66.7M - 2.33.4 514e6a882f6e ca0af62bf736 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (4m) 11s ago 4m 74.2M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (3m) 11s ago 3m 73.3M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (3m) 11s ago 3m 73.7M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:43:21.387 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (3m) 11s ago 3m 72.1M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:43:21.661 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:21 smithi062 systemd[1]: Stopping Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:21.661 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:21 smithi062 bash[58847]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager.a" found: no such container 2024-06-01T02:43:21.825 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:43:21.825 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:43:21.825 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-01T02:43:21.825 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 1 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": {}, 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:43:21.826 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 16, 2024-06-01T02:43:21.827 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 1 2024-06-01T02:43:21.827 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:43:21.827 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:43:21.971 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-mon[25095]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:21.971 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-mon[25095]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:21.971 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2951333745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:43:21.971 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-mon[29843]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:21.971 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-mon[29843]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:21.971 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2951333745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:43:21.972 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[49670]: level=info ts=2024-06-01T02:43:21.662Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:43:21.972 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:21 smithi062 podman[58876]: 2024-06-01 02:43:21.692649039 +0000 UTC m=+0.164677197 container died 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:22.178 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [], 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "1/23 daemons upgraded", 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "", 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:43:22.179 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:43:22.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:21 smithi146 ceph-mon[25945]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:22.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:21 smithi146 ceph-mon[25945]: from='client.25042 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:22.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:21 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2951333745' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:43:22.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 podman[58876]: 2024-06-01 02:43:22.103585862 +0000 UTC m=+0.575613998 container cleanup 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:22.262 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 bash[58876]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a 2024-06-01T02:43:22.263 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 bash[58925]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager.a" found: no such container 2024-06-01T02:43:22.609 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:43:22.609 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 podman[58890]: 2024-06-01 02:43:22.338406446 +0000 UTC m=+0.641506122 container remove 7ecddd10ccf3120d914e890c817e0967f891f7cc605c3ffd9ad95d3fe673e698 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:22.609 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 podman[58890]: 2024-06-01 02:43:22.405638563 +0000 UTC m=+0.708738247 volume remove 3c7181e228f9d3b4edd3e6da0a6540a2a1d1ded186b90b676a3944777a1bd5f4 2024-06-01T02:43:22.609 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-01T02:43:22.609 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 systemd[1]: Stopped Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:22.610 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 systemd[1]: Starting Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:22.705 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:22.314Z caller=manager.go:609 level=warn component="rule manager" group=pools msg="Evaluating rule failed" rule="alert: CephPoolGrowthWarning\nexpr: (predict_linear(ceph_pool_percent_used[2d], 3600 * 24 * 5) * on(pool_id) group_right()\n ceph_pool_metadata) >= 95\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.9.2\n severity: warning\n type: ceph_default\nannotations:\n description: |\n Pool '{{ $labels.name }}' will be full in less than 5 days assuming the average fill-up rate of the past 48 hours.\n summary: Pool growth rate may soon exceed it's capacity\n" err="found duplicate series for the match group {pool_id=\"1\"} on the left hand-side of the operation: [{instance=\"172.21.15.62:9283\", job=\"ceph\", pool_id=\"1\"}, {instance=\"172.21.15.146:9283\", job=\"ceph\", pool_id=\"1\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:43:22.897 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 podman[59011]: 2024-06-01 02:43:22.631805348 +0000 UTC m=+0.105881860 volume create d0383d4fe8c5b86e86112a7b864a656e5d8f84f54891383170f084746f457c17 2024-06-01T02:43:22.897 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 podman[59011]: 2024-06-01 02:43:22.54379557 +0000 UTC m=+0.017872093 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-06-01T02:43:22.897 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:22 smithi062 podman[59011]: 2024-06-01 02:43:22.715708723 +0000 UTC m=+0.189785254 container create f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:23.150 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[25095]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[25095]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[25095]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2466551402' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[29843]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[29843]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[29843]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:23.151 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:22 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2466551402' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 podman[59011]: 2024-06-01 02:43:23.05172971 +0000 UTC m=+0.525806235 container init f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 podman[59011]: 2024-06-01 02:43:23.054739116 +0000 UTC m=+0.528815629 container start f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.066Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.066Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.067Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.62 port=9094 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.069Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.094Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.094Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.097Z caller=main.go:518 msg=Listening address=:9093 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:23.097Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-06-01T02:43:23.152 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 bash[59011]: f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 2024-06-01T02:43:23.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:22 smithi146 ceph-mon[25945]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:23.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:22 smithi146 ceph-mon[25945]: from='client.25054 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:23.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:22 smithi146 ceph-mon[25945]: from='client.24704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:23.196 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:22 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2466551402' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:43:23.408 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:23 smithi062 systemd[1]: Started Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:24.112 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:24.112 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:24.112 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:24.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:23 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:23 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:24 smithi146 ceph-mon[25945]: Reconfiguring iscsi.foo.smithi062.ywlcmk (dependencies changed)... 2024-06-01T02:43:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:24 smithi146 ceph-mon[25945]: Reconfiguring daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:43:25.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:24 smithi062 ceph-mon[25095]: Reconfiguring iscsi.foo.smithi062.ywlcmk (dependencies changed)... 2024-06-01T02:43:25.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:24 smithi062 ceph-mon[25095]: Reconfiguring daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:43:25.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:24 smithi062 ceph-mon[29843]: Reconfiguring iscsi.foo.smithi062.ywlcmk (dependencies changed)... 2024-06-01T02:43:25.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:24 smithi062 ceph-mon[29843]: Reconfiguring daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:43:25.263 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:25 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:25.070Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.001047993s 2024-06-01T02:43:26.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:25 smithi062 ceph-mon[25095]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:26.164 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:25 smithi062 ceph-mon[29843]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:26.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:25 smithi146 ceph-mon[25945]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:26] "GET /metrics HTTP/1.1" 200 38151 "" "Prometheus/2.33.4" 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:26.490+0000 7f3afa1b5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Creating ceph-iscsi config... 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:26.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Traceback (most recent call last): 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return _run_code(code, main_globals, None, 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: exec(code, run_globals) 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:26.707 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Traceback (most recent call last): 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-06-01T02:43:26.708 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return self.event_loop.get_result(coro, timeout) 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return future.result(timeout) 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return self.__get_result() 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: raise self._exception 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: use_current_daemon_image=reconfig, 2024-06-01T02:43:26.709 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Creating ceph-iscsi config... 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Traceback (most recent call last): 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return _run_code(code, main_globals, None, 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:26.710 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: exec(code, run_globals) 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:26.711 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:27.904 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:27 smithi146 ceph-mon[25945]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-06-01T02:43:27.905 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-06-01T02:43:27.906 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.event_loop.get_result(coro, timeout) 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: return future.result(timeout) 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: return self.__get_result() 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: raise self._exception 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_current_daemon_image=reconfig, 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: Creating ceph-iscsi config... 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:27.907 INFO:journalctl@ceph.mon.b.smithi146.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: Traceback (most recent call last): 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: exec(code, run_globals) 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:27.908 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:27.909 INFO:journalctl@ceph.mon.b.smithi146.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:27.909 INFO:journalctl@ceph.mon.b.smithi146.stdout: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:27.909 INFO:journalctl@ceph.mon.b.smithi146.stdout: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:27.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:27 smithi146 ceph-mon[25945]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:43:27.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:27 smithi146 ceph-mon[25945]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:27.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:27 smithi146 ceph-mon[25945]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[25095]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: Creating ceph-iscsi config... 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout: Traceback (most recent call last): 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: exec(code, run_globals) 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.264 INFO:journalctl@ceph.mon.a.smithi062.stdout: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: Traceback (most recent call last): 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: return self.event_loop.get_result(coro, timeout) 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: return future.result(timeout) 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: return self.__get_result() 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: raise self._exception 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-06-01T02:43:28.265 INFO:journalctl@ceph.mon.a.smithi062.stdout: use_current_daemon_image=reconfig, 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: Creating ceph-iscsi config... 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.266 INFO:journalctl@ceph.mon.a.smithi062.stdout: Traceback (most recent call last): 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: exec(code, run_globals) 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.267 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[25095]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[25095]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[25095]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[29843]: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.c.smithi062.stdout: Creating ceph-iscsi config... 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.c.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.c.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.c.smithi062.stdout: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:28.268 INFO:journalctl@ceph.mon.c.smithi062.stdout: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: Traceback (most recent call last): 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: exec(code, run_globals) 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:28.269 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: Traceback (most recent call last): 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: return self.event_loop.get_result(coro, timeout) 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: return future.result(timeout) 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: return self.__get_result() 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-06-01T02:43:28.270 INFO:journalctl@ceph.mon.c.smithi062.stdout: raise self._exception 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: use_current_daemon_image=reconfig, 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: Creating ceph-iscsi config... 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: Traceback (most recent call last): 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:43:28.271 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: exec(code, run_globals) 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:28.272 INFO:journalctl@ceph.mon.c.smithi062.stdout: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:28.273 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[29843]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:43:28.273 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[29843]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:28.273 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:27 smithi062 ceph-mon[29843]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:43:30.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-mon[25945]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:30.204 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 systemd[1]: Stopping Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 bash[50302]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[42733]: ts=2024-06-01T02:43:29.938Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-01T02:43:30.205 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:29 smithi146 podman[50314]: 2024-06-01 02:43:29.948281178 +0000 UTC m=+0.115515302 container died ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:29 smithi062 ceph-mon[25095]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:30.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:29 smithi062 ceph-mon[29843]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:30.456 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 podman[50314]: 2024-06-01 02:43:30.304015257 +0000 UTC m=+0.471249372 container cleanup ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:30.456 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 bash[50314]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a 2024-06-01T02:43:30.456 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 bash[50339]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 podman[50328]: 2024-06-01 02:43:30.572132619 +0000 UTC m=+0.626064361 container remove ca0af62bf736a52afeeb3dd33d5a89c81ad43d1e138a45f960687a7c029864ff (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 systemd[1]: Stopped Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a.service: Consumed 1.456s CPU time. 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 systemd[1]: Starting Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 podman[50403]: 2024-06-01 02:43:30.705146124 +0000 UTC m=+0.018500088 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-01T02:43:30.911 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:30 smithi146 podman[50403]: 2024-06-01 02:43:30.806447648 +0000 UTC m=+0.119801642 container create 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:31.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:30 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 podman[50403]: 2024-06-01 02:43:31.058741998 +0000 UTC m=+0.372095950 container init 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 podman[50403]: 2024-06-01 02:43:31.06151562 +0000 UTC m=+0.374869581 container start 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.085Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.085Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.085Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.085Z caller=main.go:518 level=info host_details="(Linux 5.14.0-452.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024 x86_64 smithi146 (none))" 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.085Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.085Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.087Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-01T02:43:31.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.087Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-01T02:43:31.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.088Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-01T02:43:31.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.090Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-01T02:43:31.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.090Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.534µs 2024-06-01T02:43:31.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.090Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-01T02:43:31.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 bash[50403]: 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 2024-06-01T02:43:31.207 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 systemd[1]: Started Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:31.265 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:30 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:31.266 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:30 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:31.530 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.287Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-06-01T02:43:31.531 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.531Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-06-01T02:43:31.531 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.531Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-06-01T02:43:31.897 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:31 smithi146 systemd[1]: Stopping Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:31.897 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:31 smithi146 bash[50527]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter.b" found: no such container 2024-06-01T02:43:31.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.531Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.445µs wal_replay_duration=441.057176ms total_replay_duration=441.104046ms 2024-06-01T02:43:31.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.535Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-01T02:43:31.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.535Z caller=main.go:947 level=info msg="TSDB started" 2024-06-01T02:43:31.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.535Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-01T02:43:31.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.559Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=23.817338ms db_storage=847ns remote_storage=1.418µs web_handler=366ns query_engine=610ns scrape=161.799µs scrape_sd=100.741µs notify=10.525µs notify_sd=13.119µs rules=22.956058ms 2024-06-01T02:43:31.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:31.559Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-01T02:43:32.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-mon[25945]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:43:32.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:32.188 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:31 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:32.188 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:31 smithi146 podman[50537]: 2024-06-01 02:43:31.897241321 +0000 UTC m=+0.122806137 container died bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:43:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:31 smithi062 ceph-mon[25095]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:43:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:31 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:31 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:32.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:31 smithi062 ceph-mon[29843]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:43:32.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:31 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:32.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:31 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:32.456 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 podman[50537]: 2024-06-01 02:43:32.29490476 +0000 UTC m=+0.520469574 container cleanup bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:43:32.456 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 bash[50537]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b 2024-06-01T02:43:32.729 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 podman[50549]: 2024-06-01 02:43:32.479447139 +0000 UTC m=+0.586109883 container remove bcacd1daad1eb6f2252ae5d9dccc946f1a7ef35c8b008ce846ec26be0b6a0c81 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:43:32.729 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:43:32.729 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 bash[50565]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter.b" found: no such container 2024-06-01T02:43:32.729 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-06-01T02:43:32.730 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 systemd[1]: Stopped Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:32.730 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service: Consumed 2.703s CPU time. 2024-06-01T02:43:32.730 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 systemd[1]: Starting Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-mon[25945]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-01T02:43:33.017 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-mon[25945]: Reconfiguring daemon node-exporter.b on smithi146 2024-06-01T02:43:33.017 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 podman[50625]: 2024-06-01 02:43:32.730329244 +0000 UTC m=+0.100093900 container create c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:43:33.017 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 podman[50625]: 2024-06-01 02:43:32.647218956 +0000 UTC m=+0.016983618 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 podman[50625]: 2024-06-01 02:43:32.92624739 +0000 UTC m=+0.296012048 container init c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 podman[50625]: 2024-06-01 02:43:32.930057375 +0000 UTC m=+0.299822031 container start c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.933Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.933Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.933Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.933Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-06-01T02:43:33.018 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=arp 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=bcache 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=bonding 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=btrfs 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=conntrack 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=cpu 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-06-01T02:43:33.019 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=diskstats 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=dmi 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=edac 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=entropy 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=filefd 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=filesystem 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=hwmon 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=infiniband 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=ipvs 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=loadavg 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=mdadm 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=meminfo 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=netclass 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=netdev 2024-06-01T02:43:33.020 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=netstat 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=nfs 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=nfsd 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=nvme 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=os 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=pressure 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=rapl 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=schedstat 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=sockstat 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=softnet 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=stat 2024-06-01T02:43:33.021 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=tapestats 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=textfile 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=time 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=uname 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=vmstat 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=xfs 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:115 level=info collector=zfs 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.934Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b[50639]: ts=2024-06-01T02:43:32.935Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-06-01T02:43:33.022 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:33 smithi146 bash[50625]: c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e 2024-06-01T02:43:33.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:32 smithi062 ceph-mon[25095]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-01T02:43:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:32 smithi062 ceph-mon[25095]: Reconfiguring daemon node-exporter.b on smithi146 2024-06-01T02:43:33.263 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:43:33.073Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.004154535s 2024-06-01T02:43:33.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:32 smithi062 ceph-mon[29843]: Reconfiguring node-exporter.b (dependencies changed)... 2024-06-01T02:43:33.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:32 smithi062 ceph-mon[29843]: Reconfiguring daemon node-exporter.b on smithi146 2024-06-01T02:43:33.455 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:43:33 smithi146 systemd[1]: Started Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:34.028 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:33 smithi146 ceph-mon[25945]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:43:34.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:33 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:33 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:33 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:33 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.029 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:33 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[25095]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[29843]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:34.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:33 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:34 smithi146 ceph-mon[25945]: Reconfiguring grafana.a (dependencies changed)... 2024-06-01T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:34 smithi146 ceph-mon[25945]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-01T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:34 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:43:35.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:34 smithi146 ceph-mon[25945]: Reconfiguring daemon grafana.a on smithi146 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[25095]: Reconfiguring grafana.a (dependencies changed)... 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[25095]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[25095]: Reconfiguring daemon grafana.a on smithi146 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[29843]: Reconfiguring grafana.a (dependencies changed)... 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[29843]: Regenerating cephadm self-signed grafana TLS certificates 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-01T02:43:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:34 smithi062 ceph-mon[29843]: Reconfiguring daemon grafana.a on smithi146 2024-06-01T02:43:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:35 smithi146 ceph-mon[25945]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:36.206 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:35 smithi146 systemd[1]: Stopping Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:36.206 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:35 smithi146 bash[50809]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana.a" found: no such container 2024-06-01T02:43:36.206 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[41945]: t=2024-06-01T02:43:35+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-06-01T02:43:36.206 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 podman[50820]: 2024-06-01 02:43:36.003162694 +0000 UTC m=+0.127441867 container died 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, vcs-type=git, description=Ceph Grafana Container, architecture=x86_64, io.openshift.tags=base rhel8, release=236.1648460182, io.openshift.expose-services=, io.buildah.version=1.24.2, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, vendor=Red Hat, Inc., version=8.5, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, name=ubi8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56) 2024-06-01T02:43:36.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:35 smithi062 ceph-mon[25095]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:35 smithi062 ceph-mon[29843]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:36.618 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 podman[50820]: 2024-06-01 02:43:36.384632173 +0000 UTC m=+0.508911333 container cleanup 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , distribution-scope=public, vcs-type=git, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, architecture=x86_64, release=236.1648460182, version=8.5, name=ubi8, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-06-01T02:43:36.619 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 bash[50820]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a 2024-06-01T02:43:36.619 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 bash[50845]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana.a" found: no such container 2024-06-01T02:43:36.955 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 podman[50832]: 2024-06-01 02:43:36.619196199 +0000 UTC m=+0.616090895 container remove 587510bef419e4929252548afb975352eca9e214696ea2d2a51bec9b1cabcd09 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, architecture=x86_64, com.redhat.component=ubi8-container, release=236.1648460182, version=8.5, io.openshift.tags=base rhel8, vcs-type=git, io.openshift.expose-services=, distribution-scope=public, io.buildah.version=1.24.2, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-06-01T02:43:36.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@grafana.a.service: Deactivated successfully. 2024-06-01T02:43:36.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 systemd[1]: Stopped Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:36.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@grafana.a.service: Consumed 2.459s CPU time. 2024-06-01T02:43:36.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 systemd[1]: Starting Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:36.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 podman[50910]: 2024-06-01 02:43:36.75208212 +0000 UTC m=+0.017005671 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-06-01T02:43:36.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:36 smithi146 podman[50910]: 2024-06-01 02:43:36.861680066 +0000 UTC m=+0.126603645 container create 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, release=236.1648460182, build-date=2022-03-28T10:36:18.413762, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, description=Ceph Grafana Container, distribution-scope=public, version=8.5) 2024-06-01T02:43:37.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STOPPING 2024-06-01T02:43:37.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:43:37.242 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STOPPED 2024-06-01T02:43:37.243 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STARTING 2024-06-01T02:43:37.243 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 podman[50910]: 2024-06-01 02:43:37.0721033 +0000 UTC m=+0.337026831 container init 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, version=8.5, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vendor=Red Hat, Inc., name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.openshift.tags=base rhel8, release=236.1648460182, distribution-scope=public) 2024-06-01T02:43:37.243 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 podman[50910]: 2024-06-01 02:43:37.074857594 +0000 UTC m=+0.339781124 container start 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, release=236.1648460182, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, name=ubi8, vcs-type=git, io.openshift.tags=base rhel8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.component=ubi8-container, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, io.buildah.version=1.24.2, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8.5) 2024-06-01T02:43:37.243 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-06-01T02:43:37.243 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-06-01T02:43:37.243 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-06-01T02:43:37.244 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="App mode production" logger=settings 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="migrations completed" logger=migrator performed=0 skipped=377 duration=494.478µs 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 bash[50910]: 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 systemd[1]: Started Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-06-01T02:43:37.245 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-06-01T02:43:37.246 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-06-01T02:43:37.246 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="deleted datasource based on configuration" logger=provisioning.datasources name=Dashboard1 2024-06-01T02:43:37.246 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-06-01T02:43:37.246 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Loki uid=P8E80F9AEF21F6940 2024-06-01T02:43:37.529 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-06-01T02:43:37.530 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-06-01T02:43:37.530 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:43:37+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Serving on http://:::9283 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STARTED 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STOPPING 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STOPPED 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STARTING 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Serving on http://:::9283 2024-06-01T02:43:37.530 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STARTED 2024-06-01T02:43:37.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STOPPING 2024-06-01T02:43:37.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:43:37.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STOPPED 2024-06-01T02:43:37.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STARTING 2024-06-01T02:43:37.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Serving on http://:::9283 2024-06-01T02:43:37.531 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:43:37] ENGINE Bus STARTED 2024-06-01T02:43:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:37 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T02:43:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:43:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T02:43:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:37 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:39.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:43:39.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T02:43:39.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:43:39.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T02:43:39.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:43:39.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1386459568' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2796672348' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1350431142"}]: dispatch 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:43:39.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:38 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:39.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1386459568' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2796672348' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1350431142"}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:43:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi062.front.sepia.ceph.com:9093"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi146.front.sepia.ceph.com:9095"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi146.front.sepia.ceph.com:3000"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: Upgrade: Need to upgrade myself (mgr.x) 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1386459568' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:43:39.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2796672348' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1350431142"}]: dispatch 2024-06-01T02:43:39.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:39.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:43:39.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:43:39.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:43:39.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:38 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:39.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:39] "GET /metrics HTTP/1.1" 200 38150 "" "Prometheus/2.33.4" 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: Upgrade: Updating mgr.y 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: Deploying daemon mgr.y on smithi062 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2796672348' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1350431142"}]': finished 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: osdmap e98: 8 total, 8 up, 8 in 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/4018599218' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]: dispatch 2024-06-01T02:43:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]: dispatch 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: Upgrade: Updating mgr.y 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: Deploying daemon mgr.y on smithi062 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2796672348' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1350431142"}]': finished 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: osdmap e98: 8 total, 8 up, 8 in 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/4018599218' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]: dispatch 2024-06-01T02:43:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:39 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]: dispatch 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: Upgrade: Updating mgr.y 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: Deploying daemon mgr.y on smithi062 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2796672348' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/1350431142"}]': finished 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: osdmap e98: 8 total, 8 up, 8 in 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/4018599218' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]: dispatch 2024-06-01T02:43:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:39 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]: dispatch 2024-06-01T02:43:41.247 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]': finished 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[25095]: osdmap e99: 8 total, 8 up, 8 in 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3464786930' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]: dispatch 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]: dispatch 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[25095]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:40 smithi062 systemd[1]: Stopping Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]': finished 2024-06-01T02:43:41.248 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[29843]: osdmap e99: 8 total, 8 up, 8 in 2024-06-01T02:43:41.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3464786930' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]: dispatch 2024-06-01T02:43:41.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]: dispatch 2024-06-01T02:43:41.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:41 smithi062 ceph-mon[29843]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:41.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:41 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/313361820"}]': finished 2024-06-01T02:43:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:41 smithi146 ceph-mon[25945]: osdmap e99: 8 total, 8 up, 8 in 2024-06-01T02:43:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:41 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3464786930' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]: dispatch 2024-06-01T02:43:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:41 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]: dispatch 2024-06-01T02:43:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:41 smithi146 ceph-mon[25945]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 96 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:41.512 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:41 smithi062 podman[59873]: 2024-06-01 02:43:41.247516677 +0000 UTC m=+0.140729130 container died 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.expose-services=, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , version=8, architecture=x86_64) 2024-06-01T02:43:42.000 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:41 smithi062 podman[59873]: 2024-06-01 02:43:41.649639385 +0000 UTC m=+0.542851837 container cleanup 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, architecture=x86_64, RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-06-01T02:43:42.001 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:41 smithi062 bash[59873]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]': finished 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[25095]: osdmap e100: 8 total, 8 up, 8 in 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1835875661' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]: dispatch 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]: dispatch 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]': finished 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[29843]: osdmap e100: 8 total, 8 up, 8 in 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1835875661' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]: dispatch 2024-06-01T02:43:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]: dispatch 2024-06-01T02:43:42.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 podman[59888]: 2024-06-01 02:43:42.001437823 +0000 UTC m=+0.756673244 container remove 222c6c0c86752654f4208727c5c8fe16384381447a9b875af2880e12099acad5 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vcs-type=git) 2024-06-01T02:43:42.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:43:42.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-06-01T02:43:42.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 systemd[1]: Stopped Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:42.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service: Consumed 56.939s CPU time. 2024-06-01T02:43:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:42 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/957087882"}]': finished 2024-06-01T02:43:42.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:42 smithi146 ceph-mon[25945]: osdmap e100: 8 total, 8 up, 8 in 2024-06-01T02:43:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:42 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1835875661' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]: dispatch 2024-06-01T02:43:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:42 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]: dispatch 2024-06-01T02:43:42.710 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 systemd[1]: Starting Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:43:42.710 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 podman[60059]: 2024-06-01 02:43:42.573260002 +0000 UTC m=+0.017866813 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:43:42.710 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 podman[60059]: 2024-06-01 02:43:42.710708029 +0000 UTC m=+0.155314825 container create 65df26cb40ccddb7c473426106c2ab2412f94a4b90345324d7b0c75fac94da05 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-3d89acc) 2024-06-01T02:43:43.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]': finished 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[25095]: osdmap e101: 8 total, 8 up, 8 in 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3795668345' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/191358327"}]: dispatch 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[25095]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]': finished 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[29843]: osdmap e101: 8 total, 8 up, 8 in 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3795668345' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/191358327"}]: dispatch 2024-06-01T02:43:43.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[29843]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 podman[60059]: 2024-06-01 02:43:42.971368873 +0000 UTC m=+0.415975679 container init 65df26cb40ccddb7c473426106c2ab2412f94a4b90345324d7b0c75fac94da05 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 podman[60059]: 2024-06-01 02:43:42.974526657 +0000 UTC m=+0.419133454 container start 65df26cb40ccddb7c473426106c2ab2412f94a4b90345324d7b0c75fac94da05 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240528) 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:42 smithi062 ceph-mgr[60077]: -- 172.21.15.62:0/4208598010 <== mon.1 v2:172.21.15.62:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55779e37c5a0 con 0x55779e351000 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 bash[60059]: 65df26cb40ccddb7c473426106c2ab2412f94a4b90345324d7b0c75fac94da05 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 systemd[1]: Started Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:43:43.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:43.134+0000 7f325b490200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:43:43.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/191358327"}]': finished 2024-06-01T02:43:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-mon[25945]: osdmap e101: 8 total, 8 up, 8 in 2024-06-01T02:43:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3795668345' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/191358327"}]: dispatch 2024-06-01T02:43:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-mon[25945]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-01T02:43:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:43.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:43.347+0000 7f325b490200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:43:43.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:43.460+0000 7f325b490200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:43:43.955 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:43:44.157 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:43.820+0000 7f325b490200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:43:44.158 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:43.910+0000 7f325b490200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:43:44.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:44 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3795668345' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/191358327"}]': finished 2024-06-01T02:43:44.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:44 smithi146 ceph-mon[25945]: osdmap e102: 8 total, 8 up, 8 in 2024-06-01T02:43:44.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:44 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/131688156' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]: dispatch 2024-06-01T02:43:44.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:44 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]: dispatch 2024-06-01T02:43:44.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:44 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:44.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:44 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:44.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3795668345' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/191358327"}]': finished 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[25095]: osdmap e102: 8 total, 8 up, 8 in 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/131688156' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]: dispatch 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]: dispatch 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3795668345' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/191358327"}]': finished 2024-06-01T02:43:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[29843]: osdmap e102: 8 total, 8 up, 8 in 2024-06-01T02:43:44.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/131688156' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]: dispatch 2024-06-01T02:43:44.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]: dispatch 2024-06-01T02:43:44.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:44.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:44 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:45.453 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[25095]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]': finished 2024-06-01T02:43:45.454 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[25095]: osdmap e103: 8 total, 8 up, 8 in 2024-06-01T02:43:45.454 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1639734325' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/872011279"}]: dispatch 2024-06-01T02:43:45.454 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[25095]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-01T02:43:45.454 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:45.252+0000 7f325b490200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:43:45.454 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:45.453+0000 7f325b490200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:43:45.454 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[29843]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]': finished 2024-06-01T02:43:45.455 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[29843]: osdmap e103: 8 total, 8 up, 8 in 2024-06-01T02:43:45.455 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1639734325' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/872011279"}]: dispatch 2024-06-01T02:43:45.455 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-mon[29843]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-01T02:43:45.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:45 smithi146 ceph-mon[25945]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/51456829"}]': finished 2024-06-01T02:43:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:45 smithi146 ceph-mon[25945]: osdmap e103: 8 total, 8 up, 8 in 2024-06-01T02:43:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:45 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1639734325' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/872011279"}]: dispatch 2024-06-01T02:43:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:45 smithi146 ceph-mon[25945]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-06-01T02:43:45.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:45.637+0000 7f325b490200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:43:46.185 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:45 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:45.948+0000 7f325b490200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:43:46.185 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:46.046+0000 7f325b490200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:43:46.186 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1639734325' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/872011279"}]': finished 2024-06-01T02:43:46.186 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-mon[29843]: osdmap e104: 8 total, 8 up, 8 in 2024-06-01T02:43:46.186 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:46.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:46 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1639734325' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/872011279"}]': finished 2024-06-01T02:43:46.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:46 smithi146 ceph-mon[25945]: osdmap e104: 8 total, 8 up, 8 in 2024-06-01T02:43:46.317 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:46 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:46.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1639734325' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/872011279"}]': finished 2024-06-01T02:43:46.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-mon[25095]: osdmap e104: 8 total, 8 up, 8 in 2024-06-01T02:43:46.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:46.477 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:46.267+0000 7f325b490200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:43:46.478 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:46 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:46.477+0000 7f325b490200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:43:47.312 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:47 smithi146 ceph-mon[25945]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:47.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:47 smithi062 ceph-mon[25095]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:47.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:47 smithi062 ceph-mon[29843]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 97 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:47.513 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:47.131+0000 7f325b490200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:43:47.918 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:47.590+0000 7f325b490200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:43:47.918 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:47.697+0000 7f325b490200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:43:47.918 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:47.918+0000 7f325b490200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-01T02:43:48.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[25095]: from='client.15192 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:48.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.764 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:48.486+0000 7f325b490200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:43:48.764 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:48.588+0000 7f325b490200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:43:48.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[29843]: from='client.15192 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:48.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:48 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:48 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:48 smithi146 ceph-mon[25945]: from='client.15192 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:43:48.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:48 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:48 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:48.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:48.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:43:49.110 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:48.857+0000 7f325b490200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:43:49.110 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:49.006+0000 7f325b490200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:43:49.110 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:49.109+0000 7f325b490200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:43:49.445 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:49.237+0000 7f325b490200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:43:49.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:49 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:43:49] "GET /metrics HTTP/1.1" 200 38144 "" "Prometheus/2.33.4" 2024-06-01T02:43:49.741 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:49.577+0000 7f325b490200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:43:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-mon[29843]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:50.012 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:50.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-mon[25095]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:49 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:50.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:49 smithi146 ceph-mon[25945]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:43:50.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:49 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:50.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:49 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:50.486 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:50.229+0000 7f325b490200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:43:50.487 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:50.461+0000 7f325b490200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:43:50.487 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:50] ENGINE Bus STARTING 2024-06-01T02:43:50.762 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: CherryPy Checker: 2024-06-01T02:43:50.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: The Application mounted at '' has an empty config. 2024-06-01T02:43:50.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:43:50.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:50] ENGINE Serving on http://:::9283 2024-06-01T02:43:50.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:50] ENGINE Bus STARTED 2024-06-01T02:43:51.081 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[25095]: Standby manager daemon y restarted 2024-06-01T02:43:51.081 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[25095]: Standby manager daemon y started 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[29843]: Standby manager daemon y restarted 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[29843]: Standby manager daemon y started 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-01T02:43:51.082 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:50 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:43:51.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:50 smithi146 ceph-mon[25945]: Standby manager daemon y restarted 2024-06-01T02:43:51.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:50 smithi146 ceph-mon[25945]: Standby manager daemon y started 2024-06-01T02:43:51.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:50 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-06-01T02:43:51.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:50 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:43:51.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:50 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-06-01T02:43:51.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:50 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:43:52.093 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:51 smithi062 ceph-mon[25095]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-06-01T02:43:52.093 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:51 smithi062 ceph-mon[25095]: mgrmap e27: x(active, since 51s), standbys: y 2024-06-01T02:43:52.093 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:51 smithi062 ceph-mon[29843]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-06-01T02:43:52.093 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:51 smithi062 ceph-mon[29843]: mgrmap e27: x(active, since 51s), standbys: y 2024-06-01T02:43:52.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:51 smithi146 ceph-mon[25945]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 683 B/s rd, 0 op/s 2024-06-01T02:43:52.107 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:51 smithi146 ceph-mon[25945]: mgrmap e27: x(active, since 51s), standbys: y 2024-06-01T02:43:52.975 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:43:53.565 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:53.657 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: Reconfiguring iscsi.foo.smithi062.ywlcmk (dependencies changed)... 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[25095]: Reconfiguring daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.658 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:53.659 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: Reconfiguring iscsi.foo.smithi062.ywlcmk (dependencies changed)... 2024-06-01T02:43:53.659 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:53.659 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:53.659 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:53.659 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:53 smithi062 ceph-mon[29843]: Reconfiguring daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:43:53.697 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (31s) 6s ago 5m 14.3M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (16s) 5s ago 4m 42.3M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (16s) 6s ago 4m 74.7M - 3.5 e1d6a67b021e c9c7e4338033 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283 running (68s) 5s ago 9m 510M - 19.0.0-2946-g3d89accf b045ac545f82 03f23699819f 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (11s) 6s ago 11m 62.0M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (11m) 6s ago 11m 52.9M 2048M 17.2.0 e1d6a67b021e 74c10effbe9a 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (10m) 5s ago 10m 43.1M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (10m) 6s ago 10m 43.7M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (36s) 6s ago 5m 7884k - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (21s) 5s ago 5m 9294k - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (9m) 6s ago 9m 51.8M 2736M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (9m) 6s ago 9m 54.6M 2736M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (8m) 6s ago 8m 51.5M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:43:53.698 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (8m) 6s ago 8m 52.6M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (7m) 5s ago 7m 55.7M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (7m) 5s ago 7m 52.5M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (7m) 5s ago 7m 52.5M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (6m) 5s ago 6m 54.6M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (22s) 5s ago 6m 52.7M - 2.33.4 514e6a882f6e 72a9b8652182 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (4m) 6s ago 4m 74.4M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (4m) 5s ago 4m 73.4M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (4m) 6s ago 4m 73.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:43:53.699 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (4m) 5s ago 4m 72.2M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:43:53.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: Reconfiguring iscsi.foo.smithi062.ywlcmk (dependencies changed)... 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:43:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:53 smithi146 ceph-mon[25945]: Reconfiguring daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:43:54.112 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:43:54.112 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": {}, 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-06-01T02:43:54.113 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:43:54.114 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:43:54.114 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:43:54.459 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:43:54.459 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:43:54.459 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/23 daemons upgraded", 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mgr daemons", 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:43:54.460 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[29843]: from='client.25150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[29843]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[29843]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/1096854068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[25095]: from='client.25150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[25095]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[25095]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:54 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/1096854068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:43:54.877 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:43:54.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:54 smithi146 ceph-mon[25945]: from='client.25150 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:54 smithi146 ceph-mon[25945]: from='client.25156 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:54 smithi146 ceph-mon[25945]: from='client.25162 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:54 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1096854068' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:43:55.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:55 smithi062 ceph-mon[25095]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:55.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:55 smithi062 ceph-mon[25095]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:43:55.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:55 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/2031165459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:43:55.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:55 smithi062 ceph-mon[29843]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:55.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:55 smithi062 ceph-mon[29843]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:43:55.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:55 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/2031165459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:43:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:55 smithi146 ceph-mon[25945]: from='client.25174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:43:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:55 smithi146 ceph-mon[25945]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:43:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:55 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/2031165459' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:43:56.458 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:56.185+0000 7f3afa1b5700 -1 log_channel(cephadm) log [ERR] : cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Creating ceph-iscsi config... 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Traceback (most recent call last): 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return _run_code(code, main_globals, None, 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: exec(code, run_globals) 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:56.459 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Traceback (most recent call last): 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1114, in _check_daemons 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: self.mgr._daemon_action(daemon_spec, action=action) 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/module.py", line 2457, in _daemon_action 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: CephadmServe(self)._create_daemon(daemon_spec, reconfig=(action == 'reconfig'))) 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/module.py", line 779, in wait_async 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return self.event_loop.get_result(coro, timeout) 2024-06-01T02:43:56.460 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/ssh.py", line 136, in get_result 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return future.result(timeout) 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return self.__get_result() 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: raise self._exception 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1377, in _create_daemon 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: use_current_daemon_image=reconfig, 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1692, in _run_cephadm 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: f'cephadm exited with an error code: {code}, stderr: {err}') 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr: Reconfig daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Creating ceph-iscsi config... 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:43:56.461 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Non-zero exit code 1 from systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: Traceback (most recent call last): 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: return _run_code(code, main_globals, None, 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: exec(code, run_globals) 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1123, in deploy_daemon 2024-06-01T02:43:56.462 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:43:56.463 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: RuntimeError: Failed command: systemctl restart ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:43:56.463 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:43:56.850 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:56.850 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[25095]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-01T02:43:56.850 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-01T02:43:56.850 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[25095]: osdmap e105: 8 total, 8 up, 8 in 2024-06-01T02:43:56.851 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:56.851 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[29843]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-01T02:43:56.851 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-01T02:43:56.851 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-mon[29843]: osdmap e105: 8 total, 8 up, 8 in 2024-06-01T02:43:56.852 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:56] ENGINE Bus STOPPING 2024-06-01T02:43:56.852 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:56] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:43:56.852 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:56] ENGINE Bus STOPPED 2024-06-01T02:43:56.852 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:56] ENGINE Bus STARTING 2024-06-01T02:43:56.853 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:56] ENGINE Serving on http://:::9283 2024-06-01T02:43:56.853 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:43:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:43:56] ENGINE Bus STARTED 2024-06-01T02:43:56.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:43:56.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-mon[25945]: from='mgr.24881 172.21.15.146:0/3512002250' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-01T02:43:56.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd=[{"prefix": "mgr fail", "who": "x"}]: dispatch 2024-06-01T02:43:56.887 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-mon[25945]: osdmap e105: 8 total, 8 up, 8 in 2024-06-01T02:43:56.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:56.594+0000 7f3b67c67700 -1 mgr handle_mgr_map I was active but no longer am 2024-06-01T02:43:56.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ignoring --setuser ceph since I am not root 2024-06-01T02:43:56.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ignoring --setgroup ceph since I am not root 2024-06-01T02:43:56.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-mgr[46978]: -- 172.21.15.146:0/483510669 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5611a05e05a0 con 0x5611a05b4c00 2024-06-01T02:43:56.888 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:56.776+0000 7f08b11fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:43:57.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:56.966+0000 7f08b11fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:43:57.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:57.073+0000 7f08b11fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:43:57.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:57.397+0000 7f08b11fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:57.500+0000 7f08b11fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.24881 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: mgrmap e28: y(active, starting, since 0.396654s) 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:43:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: Manager daemon y is now available 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:57 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:57.896 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.24881 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-01T02:43:57.896 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: mgrmap e28: y(active, starting, since 0.396654s) 2024-06-01T02:43:57.896 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:43:57.896 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: Manager daemon y is now available 2024-06-01T02:43:57.897 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.24881 ' entity='mgr.x' cmd='[{"prefix": "mgr fail", "who": "x"}]': finished 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: mgrmap e28: y(active, starting, since 0.396654s) 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:43:57.898 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: Manager daemon y is now available 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:43:57.899 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:57 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:43:58.762 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:57] ENGINE Bus STARTING 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:57] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:57] ENGINE Serving on http://172.21.15.62:8765 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[25095]: [01/Jun/2024:02:43:57] ENGINE Bus STARTED 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[25095]: mgrmap e29: y(active, since 1.41521s) 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[25095]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:57] ENGINE Bus STARTING 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:57] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:57] ENGINE Serving on http://172.21.15.62:8765 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[29843]: [01/Jun/2024:02:43:57] ENGINE Bus STARTED 2024-06-01T02:43:58.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[29843]: mgrmap e29: y(active, since 1.41521s) 2024-06-01T02:43:58.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:58 smithi062 ceph-mon[29843]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:58.938 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:58.767+0000 7f08b11fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:43:58.938 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:57] ENGINE Bus STARTING 2024-06-01T02:43:58.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:57] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:43:58.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:57] ENGINE Serving on http://172.21.15.62:8765 2024-06-01T02:43:58.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-mon[25945]: [01/Jun/2024:02:43:57] ENGINE Bus STARTED 2024-06-01T02:43:58.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-mon[25945]: mgrmap e29: y(active, since 1.41521s) 2024-06-01T02:43:58.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-mon[25945]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:58.939 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:43:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:43:59.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:58.938+0000 7f08b11fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:43:59.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:59.129+0000 7f08b11fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:43:59.705 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-mon[25945]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:43:59.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-mon[25945]: mgrmap e30: y(active, since 2s) 2024-06-01T02:43:59.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:59.439+0000 7f08b11fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:43:59.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:59.545+0000 7f08b11fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:44:00.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:59 smithi062 ceph-mon[25095]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:00.003 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:43:59 smithi062 ceph-mon[25095]: mgrmap e30: y(active, since 2s) 2024-06-01T02:44:00.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:59 smithi062 ceph-mon[29843]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:00.004 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:43:59 smithi062 ceph-mon[29843]: mgrmap e30: y(active, since 2s) 2024-06-01T02:44:00.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:59.753+0000 7f08b11fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:44:00.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:43:59 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:43:59.954+0000 7f08b11fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:00.929 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:00 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:00.596+0000 7f08b11fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:44:01.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:01.074+0000 7f08b11fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:44:01.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:01.177+0000 7f08b11fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:44:01.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:01.401+0000 7f08b11fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-01T02:44:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-mon[25945]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-mon[25945]: mgrmap e31: y(active, since 4s) 2024-06-01T02:44:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[25095]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[25095]: mgrmap e31: y(active, since 4s) 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[29843]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[29843]: mgrmap e31: y(active, since 4s) 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:01 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:02.328 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:01.965+0000 7f08b11fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:44:02.328 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:02.067+0000 7f08b11fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:44:02.696 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:02.328+0000 7f08b11fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:44:02.697 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:02.471+0000 7f08b11fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:44:02.697 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:02.574+0000 7f08b11fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:44:02.697 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:02.696+0000 7f08b11fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:03.262 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:02 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.362 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:03.005+0000 7f08b11fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:44:03.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.363 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:02 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:03.634 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:03.955 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:03.634+0000 7f08b11fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:44:03.956 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:03.857+0000 7f08b11fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:44:03.956 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:44:03] ENGINE Bus STARTING 2024-06-01T02:44:04.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[25095]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:04.288 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[25095]: Standby manager daemon x started 2024-06-01T02:44:04.288 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:04.288 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[29843]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[29843]: Standby manager daemon x started 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:04.289 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:03 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-mon[25945]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-mon[25945]: Standby manager daemon x started 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/1074603467' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: CherryPy Checker: 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: The Application mounted at '' has an empty config. 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: 2024-06-01T02:44:04.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:44:03] ENGINE Serving on http://:::9283 2024-06-01T02:44:04.457 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: [01/Jun/2024:02:44:03] ENGINE Bus STARTED 2024-06-01T02:44:05.072 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:04 smithi062 ceph-mon[25095]: mgrmap e32: y(active, since 7s), standbys: x 2024-06-01T02:44:05.072 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:04 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:44:05.073 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:04 smithi062 ceph-mon[29843]: mgrmap e32: y(active, since 7s), standbys: x 2024-06-01T02:44:05.073 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:04 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:44:05.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:04 smithi146 ceph-mon[25945]: mgrmap e32: y(active, since 7s), standbys: x 2024-06-01T02:44:05.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:04 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:44:06.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[25095]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[29843]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:05 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:06.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:05 smithi146 ceph-mon[25945]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 16 op/s 2024-06-01T02:44:06.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:05 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:06.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:05 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:06.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:05 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:06.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:05 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:07.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:44:07.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:44:07.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:07.084 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.085 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.086 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.086 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:06 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:44:07.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:44:07.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:07.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:07.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:07.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:06 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:08.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:08.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:08.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:08.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:08.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:08.008 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:08.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:44:08.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:44:08.009 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:07 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3249775942' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:44:08.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3249775942' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: Reconfiguring prometheus.a (dependencies changed)... 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:44:08.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:07 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3249775942' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:44:09.191 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:08 smithi146 ceph-mon[25945]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:44:09.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:08 smithi062 ceph-mon[25095]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:44:09.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:08 smithi062 ceph-mon[29843]: Reconfiguring daemon prometheus.a on smithi146 2024-06-01T02:44:09.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[46974]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:44:09] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-06-01T02:44:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:09 smithi062 ceph-mon[25095]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:10.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:09 smithi062 ceph-mon[29843]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:10.382 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:09 smithi146 ceph-mon[25945]: pgmap v10: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:10.662 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 systemd[1]: Stopping Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:10.662 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 bash[53540]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.662Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.662Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.662Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.662Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.662Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.662Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.663Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.663Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.663Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.663Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-01T02:44:10.938 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.663Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-01T02:44:10.939 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[50418]: ts=2024-06-01T02:44:10.663Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-01T02:44:10.939 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:10 smithi146 podman[53551]: 2024-06-01 02:44:10.67330046 +0000 UTC m=+0.125164427 container died 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:44:11.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 podman[53551]: 2024-06-01 02:44:11.028601211 +0000 UTC m=+0.480465183 container cleanup 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:44:11.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 bash[53551]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a 2024-06-01T02:44:11.206 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 bash[53577]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:44:11.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 podman[53566]: 2024-06-01 02:44:11.429913309 +0000 UTC m=+0.754657600 container remove 72a9b86521820f4a931b134412683653a4334ed98f68d9ea6e680ddcae91eac8 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:44:11.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-06-01T02:44:11.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 systemd[1]: Stopped Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:11.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 systemd[1]: Starting Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:11.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 podman[53642]: 2024-06-01 02:44:11.568345447 +0000 UTC m=+0.018235928 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-06-01T02:44:11.965 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 podman[53642]: 2024-06-01 02:44:11.714008159 +0000 UTC m=+0.163898612 container create 2565089ff6fe0382e77cea2a25581ebb3118bf6e22042c34f767bb2c219c7133 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:44:11.965 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 podman[53642]: 2024-06-01 02:44:11.966453185 +0000 UTC m=+0.416343648 container init 2565089ff6fe0382e77cea2a25581ebb3118bf6e22042c34f767bb2c219c7133 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:44:11.994 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-mon[25945]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:11.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:11.995 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:11.995 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 podman[53642]: 2024-06-01 02:44:11.970038919 +0000 UTC m=+0.419929377 container start 2565089ff6fe0382e77cea2a25581ebb3118bf6e22042c34f767bb2c219c7133 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:44:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:11 smithi062 ceph-mon[25095]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:11 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:11 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:11 smithi062 ceph-mon[29843]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:11 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:11 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:12.344 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.993Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-06-01T02:44:12.344 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.994Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-06-01T02:44:12.344 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.994Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-06-01T02:44:12.344 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.994Z caller=main.go:518 level=info host_details="(Linux 5.14.0-452.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024 x86_64 smithi146 (none))" 2024-06-01T02:44:12.344 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.994Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.994Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.995Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.995Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.997Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.999Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.999Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.642µs 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:11.999Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 bash[53642]: 2565089ff6fe0382e77cea2a25581ebb3118bf6e22042c34f767bb2c219c7133 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 systemd[1]: Started Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:12.345 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.194Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=3 2024-06-01T02:44:12.669 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.449Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=3 2024-06-01T02:44:12.669 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.669Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=3 2024-06-01T02:44:12.920 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.670Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=3 maxSegment=3 2024-06-01T02:44:12.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.670Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=48.058µs wal_replay_duration=671.132439ms total_replay_duration=671.194991ms 2024-06-01T02:44:12.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.675Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-06-01T02:44:12.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.675Z caller=main.go:947 level=info msg="TSDB started" 2024-06-01T02:44:12.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.675Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-06-01T02:44:12.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.697Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=22.368906ms db_storage=870ns remote_storage=1.418µs web_handler=484ns query_engine=816ns scrape=408.008µs scrape_sd=116.563µs notify=16.864µs notify_sd=16.609µs rules=21.386105ms 2024-06-01T02:44:12.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:12.697Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-06-01T02:44:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:44:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-mon[25945]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:44:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-mon[25945]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[29843]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[29843]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:44:13.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[25095]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-01T02:44:13.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:13.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:13 smithi062 ceph-mon[25095]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 12 op/s 2024-06-01T02:44:13.673 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:13.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:14.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:14 smithi146 ceph-mon[25945]: Upgrade: Updating mgr.x 2024-06-01T02:44:14.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:14 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:44:14.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:14 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:44:14.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:14 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:14.413 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:14 smithi146 ceph-mon[25945]: Deploying daemon mgr.x on smithi146 2024-06-01T02:44:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[25095]: Upgrade: Updating mgr.x 2024-06-01T02:44:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:44:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:44:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[25095]: Deploying daemon mgr.x on smithi146 2024-06-01T02:44:14.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[29843]: Upgrade: Updating mgr.x 2024-06-01T02:44:14.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:44:14.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:44:14.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:14.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:14 smithi062 ceph-mon[29843]: Deploying daemon mgr.x on smithi146 2024-06-01T02:44:15.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:15 smithi146 ceph-mon[25945]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-06-01T02:44:15.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:15 smithi062 ceph-mon[25095]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-06-01T02:44:15.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:15 smithi062 ceph-mon[29843]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-06-01T02:44:15.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:15 smithi146 systemd[1]: Stopping Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:16.033 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:15 smithi146 podman[53991]: 2024-06-01 02:44:15.732511588 +0000 UTC m=+0.149682758 container died 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-01T02:44:16.400 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 podman[53991]: 2024-06-01 02:44:16.034050227 +0000 UTC m=+0.451221397 container cleanup 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , ceph=True) 2024-06-01T02:44:16.400 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 bash[53991]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x 2024-06-01T02:44:16.400 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 podman[54003]: 2024-06-01 02:44:16.276919152 +0000 UTC m=+0.542213371 container remove 03f23699819f2077080a4ba3112d8f3505c28be5c11f81e382768f7fab48a0cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, RELEASE=squid-3d89acc) 2024-06-01T02:44:16.401 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:44:16.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-06-01T02:44:16.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 systemd[1]: Stopped Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:16.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service: Consumed 26.690s CPU time. 2024-06-01T02:44:16.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 systemd[1]: Starting Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:17.101 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 podman[54130]: 2024-06-01 02:44:16.733422536 +0000 UTC m=+0.017730486 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:44:17.101 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:16 smithi146 podman[54130]: 2024-06-01 02:44:16.990208881 +0000 UTC m=+0.274516818 container create e90d7c11bbbfb92cc9778e9b94b488efb5e1bfb02065b7da806a84e4134e34b8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2) 2024-06-01T02:44:17.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 podman[54130]: 2024-06-01 02:44:17.205973025 +0000 UTC m=+0.490280967 container init e90d7c11bbbfb92cc9778e9b94b488efb5e1bfb02065b7da806a84e4134e34b8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-06-01T02:44:17.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 podman[54130]: 2024-06-01 02:44:17.208938839 +0000 UTC m=+0.493246788 container start e90d7c11bbbfb92cc9778e9b94b488efb5e1bfb02065b7da806a84e4134e34b8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:44:17.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-mgr[54150]: -- 172.21.15.146:0/1590073392 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5576cbc045a0 con 0x5576cbbd8800 2024-06-01T02:44:17.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 bash[54130]: e90d7c11bbbfb92cc9778e9b94b488efb5e1bfb02065b7da806a84e4134e34b8 2024-06-01T02:44:17.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 systemd[1]: Started Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:17.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:17.379+0000 7f85807df200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:44:17.834 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:17.574+0000 7f85807df200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:17.834 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:17.683+0000 7f85807df200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:44:18.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:17 smithi062 ceph-mon[25095]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-06-01T02:44:18.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:17 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:18.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:17 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:17 smithi062 ceph-mon[29843]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-06-01T02:44:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:17 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:17 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:18.124 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-mon[25945]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 603 B/s rd, 0 op/s 2024-06-01T02:44:18.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:18.125 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:17 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:18.125 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:18.036+0000 7f85807df200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:44:18.125 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:18.124+0000 7f85807df200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:44:19.100 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:18 smithi146 ceph-mon[25945]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:19.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:18 smithi062 ceph-mon[25095]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:18 smithi062 ceph-mon[29843]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:19.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:19.385+0000 7f85807df200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:44:19.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:19.587+0000 7f85807df200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:20.022 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:19 smithi146 ceph-mon[25945]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-06-01T02:44:20.023 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:19.769+0000 7f85807df200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:44:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:19 smithi062 ceph-mon[25095]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-06-01T02:44:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:19 smithi062 ceph-mon[29843]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-06-01T02:44:20.382 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:20.071+0000 7f85807df200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:44:20.382 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:20.165+0000 7f85807df200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:44:20.382 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:20.382+0000 7f85807df200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:44:20.659 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:20.585+0000 7f85807df200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:21.472 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:21.218+0000 7f85807df200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:44:21.770 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:21.660+0000 7f85807df200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:44:22.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:21 smithi146 ceph-mon[25945]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:44:22.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:21 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:22.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:21.770+0000 7f85807df200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:44:22.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:21.987+0000 7f85807df200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-01T02:44:22.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:21 smithi062 ceph-mon[25095]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:44:22.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:21 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:21 smithi062 ceph-mon[29843]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:44:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:21 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:22.852 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:22.545+0000 7f85807df200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:44:22.852 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:22.645+0000 7f85807df200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:44:23.153 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:22 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:23.154 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:22.904+0000 7f85807df200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:44:23.154 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:23.049+0000 7f85807df200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:44:23.154 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:23.153+0000 7f85807df200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:44:23.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:22 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:23.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:22 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:23.455 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:23.280+0000 7f85807df200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:23.834 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:23.606+0000 7f85807df200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:44:23.835 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:24.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-mon[25945]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:44:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:23 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:24.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:23 smithi062 ceph-mon[25095]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:44:24.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:23 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:24.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:23 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:24.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:23 smithi062 ceph-mon[29843]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:44:24.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:23 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:24.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:23 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:24.570 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:24.236+0000 7f85807df200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:44:24.571 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:24.463+0000 7f85807df200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:44:24.571 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: [01/Jun/2024:02:44:24] ENGINE Bus STARTING 2024-06-01T02:44:24.571 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: CherryPy Checker: 2024-06-01T02:44:24.571 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: The Application mounted at '' has an empty config. 2024-06-01T02:44:24.571 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:24.869 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: [01/Jun/2024:02:44:24] ENGINE Serving on http://:::9283 2024-06-01T02:44:24.870 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: [01/Jun/2024:02:44:24] ENGINE Bus STARTED 2024-06-01T02:44:25.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:44:24] "GET /metrics HTTP/1.1" 200 38234 "" "Prometheus/2.33.4" 2024-06-01T02:44:25.262 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: Standby manager daemon x restarted 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: Standby manager daemon x started 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:25.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:25 smithi146 ceph-mon[25945]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: Standby manager daemon x restarted 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: Standby manager daemon x started 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[29843]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: Standby manager daemon x restarted 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: Standby manager daemon x started 2024-06-01T02:44:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: from='mgr.? 172.21.15.146:0/2892507272' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:25 smithi062 ceph-mon[25095]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:44:25.936 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:44:25.936 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (63s) 25s ago 5m 14.3M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (49s) 4s ago 5m 45.2M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 unknown 25s ago 4m - - 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (9s) 4s ago 10m 66.1M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (43s) 25s ago 12m 477M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (12m) 25s ago 12m 53.3M 2048M 17.2.0 e1d6a67b021e 74c10effbe9a 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (10m) 4s ago 10m 44.9M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (10m) 25s ago 10m 43.9M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (68s) 25s ago 6m 8456k - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (53s) 4s ago 6m 12.8M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (10m) 25s ago 10m 52.1M 2736M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (9m) 25s ago 9m 54.9M 2736M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (9m) 25s ago 9m 50.7M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (8m) 25s ago 8m 52.8M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (8m) 4s ago 8m 56.0M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:44:25.937 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (8m) 4s ago 8m 53.1M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (7m) 4s ago 7m 52.3M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (7m) 4s ago 7m 55.5M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (14s) 4s ago 6m 50.7M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (5m) 25s ago 5m 74.6M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (5m) 4s ago 5m 74.1M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (4m) 25s ago 4m 74.1M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:44:25.938 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (4m) 4s ago 4m 73.1M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:44:26.262 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[25095]: mgrmap e33: y(active, since 28s), standbys: x 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[25095]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[25095]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[25095]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[29843]: mgrmap e33: y(active, since 28s), standbys: x 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[29843]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[29843]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:26 smithi062 ceph-mon[29843]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.353 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:44:26.353 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:44:26.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-06-01T02:44:26.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:26.353 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "mds": {}, 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 15, 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:44:26.354 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:44:26.355 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:44:26.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:26 smithi146 ceph-mon[25945]: mgrmap e33: y(active, since 28s), standbys: x 2024-06-01T02:44:26.446 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:26 smithi146 ceph-mon[25945]: from='client.15351 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:26 smithi146 ceph-mon[25945]: from='client.25249 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:26 smithi146 ceph-mon[25945]: from='client.25255 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:26.711 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:44:26.711 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:44:26.711 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:44:26.711 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "2/23 daemons upgraded", 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mon daemons", 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:44:26.712 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:44:27.133 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-01T02:44:27.133 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-01T02:44:27.133 INFO:teuthology.orchestra.run.smithi062.stdout: daemon iscsi.foo.smithi062.ywlcmk on smithi062 is in unknown state 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3825586972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: Upgrade: Setting container_image for all mgr 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.425 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: Upgrade: It appears safe to stop mon.a 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[25095]: from='client.? 172.21.15.62:0/3352152729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3825586972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:27.426 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: Upgrade: Setting container_image for all mgr 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-01T02:44:27.427 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:44:27.428 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-01T02:44:27.428 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: Upgrade: It appears safe to stop mon.a 2024-06-01T02:44:27.428 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:27.428 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:27.428 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:27 smithi062 ceph-mon[29843]: from='client.? 172.21.15.62:0/3352152729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3825586972' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: pgmap v19: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 871 B/s rd, 0 op/s 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: pgmap v20: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: Upgrade: Setting container_image for all mgr 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.x"}]': finished 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.y"}]': finished 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["a"]}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: Upgrade: It appears safe to stop mon.a 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='client.25267 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:44:27.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:27 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/3352152729' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[25095]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[25095]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[29843]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:44:28.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:28 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:28.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:28 smithi146 ceph-mon[25945]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:28.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:28 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' 2024-06-01T02:44:28.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:28 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:44:28.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:28 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:44:28.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:28 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:28.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:29.622 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:29 smithi062 ceph-mon[25095]: Upgrade: Updating mon.a 2024-06-01T02:44:29.622 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:29 smithi062 ceph-mon[25095]: Deploying daemon mon.a on smithi062 2024-06-01T02:44:29.622 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:29 smithi062 ceph-mon[25095]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-06-01T02:44:29.622 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:29 smithi062 ceph-mon[29843]: Upgrade: Updating mon.a 2024-06-01T02:44:29.622 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:29 smithi062 ceph-mon[29843]: Deploying daemon mon.a on smithi062 2024-06-01T02:44:29.623 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:29 smithi062 ceph-mon[29843]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-06-01T02:44:29.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:29 smithi146 ceph-mon[25945]: Upgrade: Updating mon.a 2024-06-01T02:44:29.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:29 smithi146 ceph-mon[25945]: Deploying daemon mon.a on smithi062 2024-06-01T02:44:29.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:29 smithi146 ceph-mon[25945]: pgmap v21: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-06-01T02:44:30.561 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:30 smithi062 systemd[1]: Stopping Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:30.835 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a[25091]: 2024-06-01T02:44:30.559+0000 7f43560a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:44:30.835 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a[25091]: 2024-06-01T02:44:30.559+0000 7f43560a7700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:30 smithi062 podman[64260]: 2024-06-01 02:44:30.601962967 +0000 UTC m=+0.147363283 container died 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0) 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:30] ENGINE Bus STOPPING 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:30] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:30] ENGINE Bus STOPPED 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:30] ENGINE Bus STARTING 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:30] ENGINE Serving on http://:::9283 2024-06-01T02:44:30.836 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:30] ENGINE Bus STARTED 2024-06-01T02:44:31.102 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:30 smithi062 podman[64260]: 2024-06-01 02:44:30.935489042 +0000 UTC m=+0.480889357 container cleanup 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vendor=Red Hat, Inc., version=8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, maintainer=Guillaume Abrioux , name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vcs-type=git) 2024-06-01T02:44:31.103 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:30 smithi062 bash[64260]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a 2024-06-01T02:44:31.368 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 podman[64285]: 2024-06-01 02:44:31.103429956 +0000 UTC m=+0.498761953 container remove 74c10effbe9a5f623ca5188858b7d037f7837a5f3fa5413ef29ddf8ba05dfa86 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.openshift.expose-services=, vcs-type=git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, version=8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_BRANCH=HEAD, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-06-01T02:44:31.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-06-01T02:44:31.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 systemd[1]: Stopped Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:31.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service: Consumed 15.389s CPU time. 2024-06-01T02:44:31.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 systemd[1]: Starting Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:32.064 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 podman[64411]: 2024-06-01 02:44:31.700057675 +0000 UTC m=+0.017341994 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:44:32.065 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:31 smithi062 podman[64411]: 2024-06-01 02:44:31.829933017 +0000 UTC m=+0.147217345 container create f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-01T02:44:32.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 podman[64411]: 2024-06-01 02:44:32.065645084 +0000 UTC m=+0.382929384 container init f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , ceph=True) 2024-06-01T02:44:32.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 podman[64411]: 2024-06-01 02:44:32.069143564 +0000 UTC m=+0.386427866 container start f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc), process ceph-mon, pid 2 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: pidfile_write: ignore empty --pid-file 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: load: jerasure load: lrc 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: RocksDB version: 7.9.2 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Git sha 0 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Compile date 2024-05-30 15:42:47 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: DB SUMMARY 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: DB Session ID: B8KPCVI14FBM9PZGP9OA 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: CURRENT file: CURRENT 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: MANIFEST file: MANIFEST-000015 size: 972 Bytes 2024-06-01T02:44:32.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000030.sst 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000028.log size: 3305221 ; 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.error_if_exists: 0 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.create_if_missing: 0 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.paranoid_checks: 1 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.env: 0x55c979e23c60 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.fs: PosixFileSystem 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.info_log: 0x55c97a771b20 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.statistics: (nil) 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.use_fsync: 0 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_log_file_size: 0 2024-06-01T02:44:32.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.allow_fallocate: 1 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.use_direct_reads: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.db_log_dir: 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.wal_dir: 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T02:44:32.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.write_buffer_manager: 0x55c97a742820 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T02:44:32.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.unordered_write: 0 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.row_cache: None 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.wal_filter: None 2024-06-01T02:44:32.518 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.two_write_queues: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.wal_compression: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.atomic_flush: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T02:44:32.519 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.log_readahead_size: 0 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_background_jobs: 2 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_background_compactions: -1 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_subcompactions: 1 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T02:44:32.520 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_open_files: -1 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_background_flushes: -1 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Compression algorithms supported: 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T02:44:32.521 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kZSTD supported: 0 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kXpressCompression supported: 0 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kZlibCompression supported: 1 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kSnappyCompression supported: 1 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kLZ4Compression supported: 1 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: kBZip2Compression supported: 0 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.merge_operator: 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_filter: None 2024-06-01T02:44:32.522 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c97a771c40) 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_type: 0 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: data_block_index_type: 0 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_shortening: 1 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: checksum: 4 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: no_block_cache: 0 2024-06-01T02:44:32.523 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache: 0x55c97a773090 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_options: 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: capacity : 536870912 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: num_shard_bits : 4 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: strict_capacity_limit : 0 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_cache_compressed: (nil) 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: persistent_cache: (nil) 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_size: 4096 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_size_deviation: 10 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_restart_interval: 16 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: index_block_restart_interval: 1 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: metadata_block_size: 4096 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: partition_filters: 0 2024-06-01T02:44:32.524 INFO:journalctl@ceph.mon.a.smithi062.stdout: use_delta_encoding: 1 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: filter_policy: bloomfilter 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: whole_key_filtering: 1 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: verify_compression: 0 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: format_version: 5 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: enable_index_compression: 1 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: block_align: 0 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: max_auto_readahead_size: 262144 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: prepopulate_block_cache: 0 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: initial_auto_readahead_size: 8192 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout: num_file_reads_for_auto_readahead: 2 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression: NoCompression 2024-06-01T02:44:32.525 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.num_levels: 7 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T02:44:32.526 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T02:44:32.527 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-01T02:44:32.528 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.arena_block_size: 1048576 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.table_properties_collectors: 2024-06-01T02:44:32.529 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.inplace_update_support: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.bloom_locality: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.max_successive_merges: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.ttl: 2592000 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-01T02:44:32.530 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.enable_blob_files: false 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.min_blob_size: 0 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.blob_file_starting_level: 0 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 12084, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-01T02:44:32.531 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9795e1c8-2d14-499b-961e-18b33168d883 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209872099557, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209872108928, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 3039582, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12085, "largest_seqno": 13468, "table_properties": {"data_size": 3032806, "index_size": 4157, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1605, "raw_key_size": 14847, "raw_average_key_size": 23, "raw_value_size": 3019291, "raw_average_value_size": 4869, "num_data_blocks": 189, "num_entries": 620, "num_filter_entries": 620, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717209872, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9795e1c8-2d14-499b-961e-18b33168d883", "db_session_id": "B8KPCVI14FBM9PZGP9OA", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209872109005, "job": 1, "event": "recovery_finished"} 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c97a862000 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: DB pointer 0x55c97a84c000 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** DB Stats ** 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:44:32.532 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: L0 1/0 2.90 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 659.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: L6 1/0 7.55 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Sum 2/0 10.45 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 659.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 659.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:44:32.533 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 659.7 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Flush(GB): cumulative 0.003, interval 0.003 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Cumulative compaction: 0.00 GB write, 233.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Interval compaction: 0.00 GB write, 233.98 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:44:32.534 INFO:journalctl@ceph.mon.a.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: Block cache BinnedLRUCache@0x55c97a773090#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: starting mon.a rank 0 at public addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] at bind addrs [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???) e3 preinit fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).mds e1 new map 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).mds e1 print_map 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: e1 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: legacy client fscid: -1 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout: No filesystems configured 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).osd e105 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-01T02:44:32.535 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:44:32.536 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:44:32.536 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).osd e105 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:44:32.536 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 ceph-mon[64430]: mon.a@-1(???).paxosservice(auth 1..22) refresh upgraded, format 0 -> 3 2024-06-01T02:44:32.536 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 bash[64411]: f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 2024-06-01T02:44:32.536 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:32 smithi062 systemd[1]: Started Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:33.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:33.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:35.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:44:34] "GET /metrics HTTP/1.1" 200 38235 "" "Prometheus/2.33.4" 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: ignoring --setuser ceph since I am not root 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: ignoring --setgroup ceph since I am not root 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mgr[54150]: -- 172.21.15.146:0/2507796259 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c334fa65a0 con 0x55c334f7b000 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:35.327+0000 7f1413404200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:44:35.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 420 B/s rd, 0 op/s 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: mon.a calling monitor election 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: monmap epoch 3 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: last_changed 2024-06-01T02:33:50.539528+0000 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: min_mon_release 17 (quincy) 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: election_strategy: 1 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:44:35.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: fsmap 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: osdmap e105: 8 total, 8 up, 8 in 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: mgrmap e33: y(active, since 38s), standbys: x 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: daemon iscsi.foo.smithi062.ywlcmk on smithi062 is in unknown state 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: from='mgr.15237 ' entity='' 2024-06-01T02:44:35.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-mon[25945]: mgrmap e34: y(active, since 38s), standbys: x 2024-06-01T02:44:35.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:44:35.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:44:35.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:44:35.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 420 B/s rd, 0 op/s 2024-06-01T02:44:35.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-06-01T02:44:35.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: mon.a calling monitor election 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: monmap epoch 3 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: last_changed 2024-06-01T02:33:50.539528+0000 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: min_mon_release 17 (quincy) 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: election_strategy: 1 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:44:35.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: fsmap 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: osdmap e105: 8 total, 8 up, 8 in 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: mgrmap e33: y(active, since 38s), standbys: x 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: daemon iscsi.foo.smithi062.ywlcmk on smithi062 is in unknown state 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: from='mgr.15237 ' entity='' 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[29843]: mgrmap e34: y(active, since 38s), standbys: x 2024-06-01T02:44:35.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: from='mgr.15237 172.21.15.62:0/2771506098' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 420 B/s rd, 0 op/s 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 0 op/s 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: mon.a calling monitor election 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: monmap epoch 3 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: last_changed 2024-06-01T02:33:50.539528+0000 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: min_mon_release 17 (quincy) 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: election_strategy: 1 2024-06-01T02:44:35.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: fsmap 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: osdmap e105: 8 total, 8 up, 8 in 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: mgrmap e33: y(active, since 38s), standbys: x 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s) 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: daemon iscsi.foo.smithi062.ywlcmk on smithi062 is in unknown state 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: from='mgr.15237 ' entity='' 2024-06-01T02:44:35.517 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mon[64430]: mgrmap e34: y(active, since 38s), standbys: x 2024-06-01T02:44:35.518 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ignoring --setuser ceph since I am not root 2024-06-01T02:44:35.518 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ignoring --setgroup ceph since I am not root 2024-06-01T02:44:35.518 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-mgr[60077]: -- 172.21.15.62:0/1116047271 <== mon.1 v2:172.21.15.62:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560d30c2c5a0 con 0x560d30c00800 2024-06-01T02:44:35.518 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:35.356+0000 7f1586556200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-01T02:44:35.927 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:35.513+0000 7f1413404200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:35.927 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:35.616+0000 7f1413404200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:44:35.928 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:35.927+0000 7f1413404200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:44:35.993 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:35.559+0000 7f1586556200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:35.993 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:35.668+0000 7f1586556200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-01T02:44:35.993 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:35.993+0000 7f1586556200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-01T02:44:36.205 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:36.013+0000 7f1413404200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:44:36.262 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:36 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:36.084+0000 7f1586556200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-01T02:44:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:37 smithi146 ceph-mon[25945]: mgrmap e35: y(active, since 39s), standbys: x 2024-06-01T02:44:37.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:37.221+0000 7f1413404200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:44:37.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:37.393+0000 7f1413404200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:37.512 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:37 smithi062 ceph-mon[64430]: mgrmap e35: y(active, since 39s), standbys: x 2024-06-01T02:44:37.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:37 smithi062 ceph-mon[29843]: mgrmap e35: y(active, since 39s), standbys: x 2024-06-01T02:44:37.513 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:37 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:37.359+0000 7f1586556200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-01T02:44:37.867 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:37.567+0000 7f1413404200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:44:38.012 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:37 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:37.540+0000 7f1586556200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:38.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:37 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:37.720+0000 7f1586556200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-01T02:44:38.173 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:37.867+0000 7f1413404200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:44:38.174 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:37.963+0000 7f1413404200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:44:38.269 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:38 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:38.027+0000 7f1586556200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-01T02:44:38.269 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:38 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:38.128+0000 7f1586556200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-01T02:44:38.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:38.173+0000 7f1413404200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:44:38.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:38.373+0000 7f1413404200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:38.559 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:38 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:38.346+0000 7f1586556200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-01T02:44:38.955 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:38.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:39.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:38 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:38.559+0000 7f1586556200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-01T02:44:39.417 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:38.985+0000 7f1413404200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:44:39.512 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:39 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:39.209+0000 7f1586556200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-01T02:44:39.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:39.417+0000 7f1413404200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:44:39.706 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:39.517+0000 7f1413404200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:44:39.978 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:39 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:39.660+0000 7f1586556200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-01T02:44:39.978 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:39 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:39.763+0000 7f1586556200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-01T02:44:40.206 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:39.723+0000 7f1413404200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-01T02:44:40.262 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:39 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:39.978+0000 7f1586556200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-01T02:44:40.605 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:40.258+0000 7f1413404200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:44:40.605 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:40.354+0000 7f1413404200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:44:40.898 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:40.537+0000 7f1586556200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-01T02:44:40.898 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:40.636+0000 7f1586556200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-01T02:44:40.956 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:40.605+0000 7f1413404200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:44:40.956 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:40.743+0000 7f1413404200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:44:40.956 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:40.842+0000 7f1413404200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:44:41.262 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:40.898+0000 7f1586556200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-01T02:44:41.263 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:41 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:41.041+0000 7f1586556200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-01T02:44:41.263 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:41 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:41.143+0000 7f1586556200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-01T02:44:41.268 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:40.961+0000 7f1413404200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:41.586 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:41 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:41.269+0000 7f1586556200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-01T02:44:41.705 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:41 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:41.268+0000 7f1413404200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:44:42.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:41 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:41.586+0000 7f1586556200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-01T02:44:42.172 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:41 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:41.885+0000 7f1413404200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:44:42.172 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:42.108+0000 7f1413404200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:44:42.173 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: [01/Jun/2024:02:44:42] ENGINE Bus STARTING 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-mon[25945]: Standby manager daemon x restarted 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-mon[25945]: Standby manager daemon x started 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-mon[25945]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: CherryPy Checker: 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: The Application mounted at '' has an empty config. 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: 2024-06-01T02:44:42.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: [01/Jun/2024:02:44:42] ENGINE Serving on http://:::9283 2024-06-01T02:44:42.457 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:44:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x[54146]: [01/Jun/2024:02:44:42] ENGINE Bus STARTED 2024-06-01T02:44:42.470 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:42.237+0000 7f1586556200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[64430]: Standby manager daemon x restarted 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[64430]: Standby manager daemon x started 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[64430]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[64430]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[64430]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[64430]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[29843]: Standby manager daemon x restarted 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[29843]: Standby manager daemon x started 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-06-01T02:44:42.471 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-mon[29843]: from='mgr.? 172.21.15.146:0/2082061530' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:42.470+0000 7f1586556200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:42] ENGINE Bus STARTING 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: CherryPy Checker: 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: The Application mounted at '' has an empty config. 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:42] ENGINE Serving on http://:::9283 2024-06-01T02:44:42.766 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:44:42] ENGINE Bus STARTED 2024-06-01T02:44:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: mgrmap e36: y(active, since 45s), standbys: x 2024-06-01T02:44:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: Active manager daemon y restarted 2024-06-01T02:44:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: Activating manager daemon y 2024-06-01T02:44:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:44:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: mgrmap e37: y(active, starting, since 0.00683425s), standbys: x 2024-06-01T02:44:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:44:43.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:44:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:44:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:44:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: Manager daemon y is now available 2024-06-01T02:44:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:44:43.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: mgrmap e36: y(active, since 45s), standbys: x 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: Active manager daemon y restarted 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: Activating manager daemon y 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: mgrmap e37: y(active, starting, since 0.00683425s), standbys: x 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:44:43.488 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: Manager daemon y is now available 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:44:43.489 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:44:43.484+0000 7f15439f1700 -1 mgr.server handle_report got status from non-daemon mon.a 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: mgrmap e36: y(active, since 45s), standbys: x 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: Active manager daemon y restarted 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: Activating manager daemon y 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: mgrmap e37: y(active, starting, since 0.00683425s), standbys: x 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:44:43.490 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-01T02:44:43.491 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: Manager daemon y is now available 2024-06-01T02:44:43.492 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:44:43.492 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-06-01T02:44:43.492 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:43 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-06-01T02:44:43.784 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:43.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:44.576 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: [01/Jun/2024:02:44:43] ENGINE Bus STARTING 2024-06-01T02:44:44.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: [01/Jun/2024:02:44:43] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:44:44.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: mgrmap e38: y(active, since 1.01162s), standbys: x 2024-06-01T02:44:44.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:44.577 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:44.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: [01/Jun/2024:02:44:43] ENGINE Serving on http://172.21.15.62:8765 2024-06-01T02:44:44.578 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:44 smithi146 ceph-mon[25945]: [01/Jun/2024:02:44:43] ENGINE Bus STARTED 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:44:44] "GET /metrics HTTP/1.1" 200 35443 "" "Prometheus/2.33.4" 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: [01/Jun/2024:02:44:43] ENGINE Bus STARTING 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: [01/Jun/2024:02:44:43] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: mgrmap e38: y(active, since 1.01162s), standbys: x 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: [01/Jun/2024:02:44:43] ENGINE Serving on http://172.21.15.62:8765 2024-06-01T02:44:44.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[64430]: [01/Jun/2024:02:44:43] ENGINE Bus STARTED 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: [01/Jun/2024:02:44:43] ENGINE Bus STARTING 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: [01/Jun/2024:02:44:43] ENGINE Serving on https://172.21.15.62:7150 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: mgrmap e38: y(active, since 1.01162s), standbys: x 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: pgmap v3: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: [01/Jun/2024:02:44:43] ENGINE Serving on http://172.21.15.62:8765 2024-06-01T02:44:44.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:44 smithi062 ceph-mon[29843]: [01/Jun/2024:02:44:43] ENGINE Bus STARTED 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[64430]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[64430]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[64430]: mgrmap e39: y(active, since 2s), standbys: x 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[29843]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[29843]: Cluster is now healthy 2024-06-01T02:44:45.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[29843]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:45.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:45 smithi062 ceph-mon[29843]: mgrmap e39: y(active, since 2s), standbys: x 2024-06-01T02:44:45.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:45 smithi146 ceph-mon[25945]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-06-01T02:44:45.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:45 smithi146 ceph-mon[25945]: Cluster is now healthy 2024-06-01T02:44:45.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:45 smithi146 ceph-mon[25945]: pgmap v4: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:45.916 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:45 smithi146 ceph-mon[25945]: mgrmap e39: y(active, since 2s), standbys: x 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[64430]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[64430]: mgrmap e40: y(active, since 4s), standbys: x 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[29843]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:47.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[29843]: mgrmap e40: y(active, since 4s), standbys: x 2024-06-01T02:44:47.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:47 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:47 smithi146 ceph-mon[25945]: pgmap v5: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:47 smithi146 ceph-mon[25945]: mgrmap e40: y(active, since 4s), standbys: x 2024-06-01T02:44:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:47 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:47 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:47 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:47 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:48.945 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[64430]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:49.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:49 smithi062 ceph-mon[29843]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:49.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:49 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:49 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:49 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:49 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:49.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:49 smithi146 ceph-mon[25945]: pgmap v6: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:44:51.812 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:51 smithi146 ceph-mon[25945]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-01T02:44:52.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:51 smithi062 ceph-mon[64430]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-01T02:44:52.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:51 smithi062 ceph-mon[29843]: pgmap v7: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-06-01T02:44:52.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:52.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:52.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:52.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:52.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:44:52.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:44:52.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:52.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:52.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:52.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:52 smithi146 ceph-mon[25945]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:53.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[29843]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: Updating smithi062:/etc/ceph/ceph.conf 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: Updating smithi146:/etc/ceph/ceph.conf 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: Updating smithi062:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:53.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:52 smithi062 ceph-mon[64430]: Updating smithi146:/etc/ceph/ceph.client.admin.keyring 2024-06-01T02:44:53.687 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:53.687 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:53.687 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-01T02:44:53.687 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.687 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.687 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[64430]: Upgrade: It appears safe to stop mon.c 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.688 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-01T02:44:53.689 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:53 smithi062 ceph-mon[29843]: Upgrade: It appears safe to stop mon.c 2024-06-01T02:44:53.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: Updating smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: Updating smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: pgmap v8: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:44:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:44:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:44:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["c"]}]: dispatch 2024-06-01T02:44:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:53 smithi146 ceph-mon[25945]: Upgrade: It appears safe to stop mon.c 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:44:54] "GET /metrics HTTP/1.1" 200 35443 "" "Prometheus/2.33.4" 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:54.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[29843]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:54.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:54.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:44:54.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:44:54.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:54 smithi062 ceph-mon[29843]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:54 smithi146 ceph-mon[25945]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:44:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:54 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:44:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:54 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:44:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:54 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:44:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:54 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:44:55.755 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:55 smithi062 ceph-mon[64430]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:55.755 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:55 smithi062 ceph-mon[64430]: Upgrade: Updating mon.c 2024-06-01T02:44:55.755 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:44:55 smithi062 ceph-mon[64430]: Deploying daemon mon.c on smithi062 2024-06-01T02:44:55.755 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:55 smithi062 ceph-mon[29843]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:55.755 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:55 smithi062 ceph-mon[29843]: Upgrade: Updating mon.c 2024-06-01T02:44:55.755 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:55 smithi062 ceph-mon[29843]: Deploying daemon mon.c on smithi062 2024-06-01T02:44:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:55 smithi146 ceph-mon[25945]: pgmap v9: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-06-01T02:44:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:55 smithi146 ceph-mon[25945]: Upgrade: Updating mon.c 2024-06-01T02:44:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:44:55 smithi146 ceph-mon[25945]: Deploying daemon mon.c on smithi062 2024-06-01T02:44:56.512 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:56 smithi062 systemd[1]: Stopping Ceph mon.c for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:56.993 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c[29839]: 2024-06-01T02:44:56.552+0000 7f60bf441700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:44:56.994 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c[29839]: 2024-06-01T02:44:56.552+0000 7f60bf441700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-06-01T02:44:56.994 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:56 smithi062 podman[66652]: 2024-06-01 02:44:56.777895601 +0000 UTC m=+0.322804421 container died 7e7d2419998caee4796d9689df8414f6727c2cf9bfc3fb59d25292d91c321747 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, release=754, vcs-type=git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container) 2024-06-01T02:44:57.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 podman[66652]: 2024-06-01 02:44:57.118404489 +0000 UTC m=+0.663313318 container cleanup 7e7d2419998caee4796d9689df8414f6727c2cf9bfc3fb59d25292d91c321747 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, ceph=True, name=centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-06-01T02:44:57.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 bash[66652]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c 2024-06-01T02:44:57.543 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:44:57.595 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 podman[66664]: 2024-06-01 02:44:57.328022024 +0000 UTC m=+0.548398364 container remove 7e7d2419998caee4796d9689df8414f6727c2cf9bfc3fb59d25292d91c321747 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-06-01T02:44:57.595 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.c.service: Deactivated successfully. 2024-06-01T02:44:57.595 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 systemd[1]: Stopped Ceph mon.c for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:57.595 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.c.service: Consumed 8.826s CPU time. 2024-06-01T02:44:58.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:57 smithi062 systemd[1]: Starting Ceph mon.c for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:44:58.258 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 podman[66840]: 2024-06-01 02:44:58.003943933 +0000 UTC m=+0.112188853 container create 7ee52beadd88d9530cdc4e89d21c77d5eca51af90de3fb2fb4c4d32046189bc7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:44:58.259 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 podman[66840]: 2024-06-01 02:44:57.910090179 +0000 UTC m=+0.018335121 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (95s) 11s ago 6m 14.3M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (81s) 11s ago 6m 45.4M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (51s) 11s ago 5m 50.7M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (41s) 11s ago 10m 449M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (75s) 11s ago 12m 476M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:44:58.259 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (26s) 11s ago 12m 29.7M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (11m) 11s ago 11m 49.7M 2048M 17.2.0 e1d6a67b021e 749ff523ad17 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (11m) 11s ago 11m 49.2M 2048M 17.2.0 e1d6a67b021e 7e7d2419998c 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (100s) 11s ago 7m 12.5M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (85s) 11s ago 6m 15.2M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (10m) 11s ago 10m 53.1M 2736M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (10m) 11s ago 10m 55.5M 2736M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (9m) 11s ago 9m 51.1M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (9m) 11s ago 9m 53.1M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (8m) 11s ago 8m 56.4M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (8m) 11s ago 8m 53.3M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (8m) 11s ago 8m 52.5M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:44:58.260 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (7m) 11s ago 7m 55.8M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:44:58.261 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (46s) 11s ago 7m 56.7M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:44:58.261 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (5m) 11s ago 5m 75.5M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:44:58.261 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (5m) 11s ago 5m 74.5M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:44:58.261 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (5m) 11s ago 5m 74.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:44:58.261 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (5m) 11s ago 5m 73.5M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:44:58.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 podman[66840]: 2024-06-01 02:44:58.264968234 +0000 UTC m=+0.373213167 container init 7ee52beadd88d9530cdc4e89d21c77d5eca51af90de3fb2fb4c4d32046189bc7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 podman[66840]: 2024-06-01 02:44:58.26869574 +0000 UTC m=+0.376940666 container start 7ee52beadd88d9530cdc4e89d21c77d5eca51af90de3fb2fb4c4d32046189bc7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc), process ceph-mon, pid 2 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: pidfile_write: ignore empty --pid-file 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: load: jerasure load: lrc 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: RocksDB version: 7.9.2 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Git sha 0 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Compile date 2024-05-30 15:42:47 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: DB SUMMARY 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: DB Session ID: NF7987KXYKF6UOLLBHR0 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: CURRENT file: CURRENT 2024-06-01T02:44:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 1, files: 000027.sst 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000025.log size: 287707 ; 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.error_if_exists: 0 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.create_if_missing: 0 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.paranoid_checks: 1 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-01T02:44:58.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.env: 0x55c92e218c60 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.fs: PosixFileSystem 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.info_log: 0x55c92fd6bb20 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.statistics: (nil) 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.use_fsync: 0 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_log_file_size: 0 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.allow_fallocate: 1 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T02:44:58.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.use_direct_reads: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.db_log_dir: 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.wal_dir: 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.write_buffer_manager: 0x55c92fd3c820 2024-06-01T02:44:58.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.unordered_write: 0 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.row_cache: None 2024-06-01T02:44:58.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.wal_filter: None 2024-06-01T02:44:58.519 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.two_write_queues: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.wal_compression: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.atomic_flush: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.log_readahead_size: 0 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T02:44:58.520 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_background_jobs: 2 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_background_compactions: -1 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_subcompactions: 1 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T02:44:58.521 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_open_files: -1 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_background_flushes: -1 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Compression algorithms supported: 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kZSTD supported: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kXpressCompression supported: 0 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T02:44:58.522 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kZlibCompression supported: 1 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kSnappyCompression supported: 1 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kLZ4Compression supported: 1 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: kBZip2Compression supported: 0 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.merge_operator: 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_filter: None 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T02:44:58.523 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c92fd6bc40) 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: cache_index_and_filter_blocks: 1 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: pin_top_level_index_and_filter: 1 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: index_type: 0 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: data_block_index_type: 0 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: index_shortening: 1 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: checksum: 4 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: no_block_cache: 0 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_cache: 0x55c92fd6d090 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_cache_name: BinnedLRUCache 2024-06-01T02:44:58.524 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_cache_options: 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: capacity : 536870912 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: num_shard_bits : 4 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: strict_capacity_limit : 0 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: high_pri_pool_ratio: 0.000 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_cache_compressed: (nil) 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: persistent_cache: (nil) 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_size: 4096 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_size_deviation: 10 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_restart_interval: 16 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: index_block_restart_interval: 1 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: metadata_block_size: 4096 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: partition_filters: 0 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: use_delta_encoding: 1 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: filter_policy: bloomfilter 2024-06-01T02:44:58.525 INFO:journalctl@ceph.mon.c.smithi062.stdout: whole_key_filtering: 1 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: verify_compression: 0 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: read_amp_bytes_per_bit: 0 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: format_version: 5 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: enable_index_compression: 1 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: block_align: 0 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: max_auto_readahead_size: 262144 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: prepopulate_block_cache: 0 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: initial_auto_readahead_size: 8192 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout: num_file_reads_for_auto_readahead: 2 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression: NoCompression 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T02:44:58.526 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.num_levels: 7 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-01T02:44:58.527 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T02:44:58.528 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-01T02:44:58.529 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.arena_block_size: 1048576 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T02:44:58.530 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.table_properties_collectors: 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.inplace_update_support: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.bloom_locality: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.max_successive_merges: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.ttl: 2592000 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T02:44:58.531 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.enable_blob_files: false 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.min_blob_size: 0 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.blob_file_starting_level: 0 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-01T02:44:58.532 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14927, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 513e0887-52cb-4f70-9253-9fc07efee5b9 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209898299659, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209898301339, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 191885, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14932, "largest_seqno": 15054, "table_properties": {"data_size": 190217, "index_size": 464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 197, "raw_key_size": 1930, "raw_average_key_size": 26, "raw_value_size": 188480, "raw_average_value_size": 2617, "num_data_blocks": 19, "num_entries": 72, "num_filter_entries": 72, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717209898, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "513e0887-52cb-4f70-9253-9fc07efee5b9", "db_session_id": "NF7987KXYKF6UOLLBHR0", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209898301428, "job": 1, "event": "recovery_finished"} 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T02:44:58.533 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c92fe5c000 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: DB pointer 0x55c92fe46000 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: ** DB Stats ** 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:44:58.534 INFO:journalctl@ceph.mon.c.smithi062.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: L0 1/0 187.39 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 194.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: L6 1/0 11.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: Sum 2/0 11.42 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 194.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 194.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: ** Compaction Stats [default] ** 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 194.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: 2024-06-01T02:44:58.535 INFO:journalctl@ceph.mon.c.smithi062.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: Cumulative compaction: 0.00 GB write, 46.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: Interval compaction: 0.00 GB write, 46.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: Block cache BinnedLRUCache@0x55c92fd6d090#2 capacity: 512.00 MB usage: 0.78 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.23 KB,4.47035e-05%) IndexBlock(1,0.55 KB,0.000104308%) Misc(1,0.00 KB,0%) 2024-06-01T02:44:58.536 INFO:journalctl@ceph.mon.c.smithi062.stdout: 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: starting mon.c rank 1 at public addrs [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] at bind addrs [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???) e3 preinit fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).mds e1 new map 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).mds e1 print_map 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: e1 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: legacy client fscid: -1 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: 2024-06-01T02:44:58.537 INFO:journalctl@ceph.mon.c.smithi062.stdout: No filesystems configured 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 ceph-mon[66884]: mon.c@-1(???).paxosservice(auth 1..23) refresh upgraded, format 0 -> 3 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 bash[66840]: 7ee52beadd88d9530cdc4e89d21c77d5eca51af90de3fb2fb4c4d32046189bc7 2024-06-01T02:44:58.538 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:44:58 smithi062 systemd[1]: Started Ceph mon.c for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:44:58.686 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 1 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:44:58.687 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:44:58.688 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:44:58.688 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14, 2024-06-01T02:44:58.688 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:44:58.688 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:44:58.688 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:44:58.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:44:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:44:58.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:44:59.035 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "3/23 daemons upgraded", 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mon daemons", 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:44:59.036 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:44:59.454 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: mon.c calling monitor election 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/546909970' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:00.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='client.? 172.21.15.62:0/1602165503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: mon.a calling monitor election 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: monmap epoch 3 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: last_changed 2024-06-01T02:33:50.539528+0000 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: min_mon_release 17 (quincy) 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: election_strategy: 1 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: fsmap 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:45:00.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: mgrmap e40: y(active, since 17s), standbys: x 2024-06-01T02:45:00.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: overall HEALTH_OK 2024-06-01T02:45:00.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:00.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:00.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:00.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:00 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: mon.c calling monitor election 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/546909970' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:01.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1602165503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: mon.a calling monitor election 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: monmap epoch 3 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: last_changed 2024-06-01T02:33:50.539528+0000 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: min_mon_release 17 (quincy) 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: election_strategy: 1 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:45:01.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: fsmap 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: mgrmap e40: y(active, since 17s), standbys: x 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: overall HEALTH_OK 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='client.34137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='client.34143 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='client.34149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: pgmap v11: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: mon.c calling monitor election 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/546909970' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='client.34161 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1602165503' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: mon.a calling monitor election 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: monmap epoch 3 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: last_changed 2024-06-01T02:33:50.539528+0000 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: min_mon_release 17 (quincy) 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: election_strategy: 1 2024-06-01T02:45:01.016 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: fsmap 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: mgrmap e40: y(active, since 17s), standbys: x 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: overall HEALTH_OK 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:01.017 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:01 smithi146 ceph-mon[25945]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:45:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:01 smithi062 ceph-mon[64430]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:45:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:01 smithi062 ceph-mon[66884]: pgmap v12: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-06-01T02:45:03.686 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:03 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:45:03.310+0000 7f15439f1700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-01T02:45:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:03 smithi146 ceph-mon[25945]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:03.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:03.978 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:03 smithi062 ceph-mon[64430]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:03.979 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:03 smithi062 ceph-mon[66884]: pgmap v13: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:04.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:04 smithi146 ceph-mon[25945]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:04.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:04 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:04.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:04 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:45:04] "GET /metrics HTTP/1.1" 200 38187 "" "Prometheus/2.33.4" 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:04 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:05.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:05 smithi146 ceph-mon[25945]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:05.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:05 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:05.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:05 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:06.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:05 smithi062 ceph-mon[64430]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:06.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:05 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:06.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:05 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:06.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:05 smithi062 ceph-mon[66884]: pgmap v14: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:06.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:05 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:06.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:05 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:07.955 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:07 smithi146 ceph-mon[25945]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:08.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:07 smithi062 ceph-mon[64430]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:08.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:07 smithi062 ceph-mon[66884]: pgmap v15: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:08.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:08.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:45:10.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:10.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:45:10.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-01T02:45:10.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:09 smithi146 ceph-mon[25945]: Upgrade: It appears safe to stop mon.b 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:45:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[64430]: Upgrade: It appears safe to stop mon.b 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: pgmap v16: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:45:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:10.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-06-01T02:45:10.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon ok-to-stop", "ids": ["b"]}]: dispatch 2024-06-01T02:45:10.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:09 smithi062 ceph-mon[66884]: Upgrade: It appears safe to stop mon.b 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: Upgrade: Updating mon.b 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:11.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:11 smithi146 ceph-mon[25945]: Deploying daemon mon.b on smithi146 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: Upgrade: Updating mon.b 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:12.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[66884]: Deploying daemon mon.b on smithi146 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: pgmap v17: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: Upgrade: Updating mon.b 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:12.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:11 smithi062 ceph-mon[64430]: Deploying daemon mon.b on smithi146 2024-06-01T02:45:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:12 smithi146 systemd[1]: Stopping Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:45:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:12 smithi146 ceph-mon[25945]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:13.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:13.050 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b[25928]: 2024-06-01T02:45:12.795+0000 7ff2cdfc8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:45:13.051 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b[25928]: 2024-06-01T02:45:12.795+0000 7ff2cdfc8700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-06-01T02:45:13.304 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 podman[57549]: 2024-06-01 02:45:13.051724906 +0000 UTC m=+0.495012083 container died 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, name=centos-stream, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.component=centos-stream-container, GIT_CLEAN=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, ceph=True, vendor=Red Hat, Inc.) 2024-06-01T02:45:13.680 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:13.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 podman[57549]: 2024-06-01 02:45:13.412226861 +0000 UTC m=+0.855514030 container cleanup 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, release=754, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-06-01T02:45:13.680 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 bash[57549]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b 2024-06-01T02:45:13.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 podman[57561]: 2024-06-01 02:45:13.680377606 +0000 UTC m=+0.621346867 container remove 749ff523ad17c9960698e3c02390e8e36445223a4d56a25f1eb9c4ae5664b31a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, version=8, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, release=754) 2024-06-01T02:45:13.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.b.service: Deactivated successfully. 2024-06-01T02:45:13.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 systemd[1]: Stopped Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:45:13.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:13 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.b.service: Consumed 9.440s CPU time. 2024-06-01T02:45:14.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 systemd[1]: Starting Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:45:14.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 podman[57690]: 2024-06-01 02:45:14.165222917 +0000 UTC m=+0.017782167 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:45:14.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 podman[57690]: 2024-06-01 02:45:14.289964448 +0000 UTC m=+0.142523683 container create b8af55737ea256365dfbbb37fa45b92155a9266147813e3cf3c6f6167216baa9 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, RELEASE=squid-3d89acc, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 podman[57690]: 2024-06-01 02:45:14.54268813 +0000 UTC m=+0.395247360 container init b8af55737ea256365dfbbb37fa45b92155a9266147813e3cf3c6f6167216baa9 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc) 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 podman[57690]: 2024-06-01 02:45:14.546172521 +0000 UTC m=+0.398731757 container start b8af55737ea256365dfbbb37fa45b92155a9266147813e3cf3c6f6167216baa9 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True) 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: set uid:gid to 167:167 (ceph:ceph) 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc), process ceph-mon, pid 2 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: pidfile_write: ignore empty --pid-file 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: load: jerasure load: lrc 2024-06-01T02:45:14.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: RocksDB version: 7.9.2 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Git sha 0 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Compile date 2024-05-30 15:42:47 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: DB SUMMARY 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: DB Session ID: JWNOY5W18H88YJU7B7NG 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: CURRENT file: CURRENT 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: IDENTITY file: IDENTITY 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: MANIFEST file: MANIFEST-000009 size: 1084 Bytes 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 1, files: 000027.sst 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000025.log size: 1028414 ; 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.error_if_exists: 0 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.create_if_missing: 0 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.paranoid_checks: 1 2024-06-01T02:45:14.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.flush_verify_memtable_count: 1 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.env: 0x55992a275c60 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.fs: PosixFileSystem 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.info_log: 0x55992b5bbb20 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_file_opening_threads: 16 2024-06-01T02:45:14.958 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.statistics: (nil) 2024-06-01T02:45:14.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.use_fsync: 0 2024-06-01T02:45:14.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_log_file_size: 0 2024-06-01T02:45:14.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-01T02:45:14.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-01T02:45:14.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.keep_log_file_num: 1000 2024-06-01T02:45:14.959 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.recycle_log_file_num: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.allow_fallocate: 1 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.allow_mmap_reads: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.allow_mmap_writes: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.use_direct_reads: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.create_missing_column_families: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.db_log_dir: 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.wal_dir: 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-01T02:45:14.960 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.advise_random_on_open: 1 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.db_write_buffer_size: 0 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.write_buffer_manager: 0x55992b58c820 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.rate_limiter: (nil) 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.wal_recovery_mode: 2 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.enable_thread_tracking: 0 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.enable_pipelined_write: 0 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.unordered_write: 0 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-01T02:45:14.961 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.row_cache: None 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.wal_filter: None 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.allow_ingest_behind: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.two_write_queues: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.manual_wal_flush: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.wal_compression: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.atomic_flush: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.log_readahead_size: 0 2024-06-01T02:45:14.962 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.best_efforts_recovery: 0 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.allow_data_in_errors: 0 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.db_host_id: __hostname__ 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.enforce_single_del_contracts: true 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_background_jobs: 2 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_background_compactions: -1 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_subcompactions: 1 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-01T02:45:14.963 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_total_wal_size: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_open_files: -1 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bytes_per_sync: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_readahead_size: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_background_flushes: -1 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Compression algorithms supported: 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kZSTD supported: 0 2024-06-01T02:45:14.964 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kXpressCompression supported: 0 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kLZ4HCCompression supported: 1 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kZlibCompression supported: 1 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kSnappyCompression supported: 1 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kLZ4Compression supported: 1 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: kBZip2Compression supported: 0 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: DMutex implementation: pthread_mutex_t 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.merge_operator: 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_filter: None 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_filter_factory: None 2024-06-01T02:45:14.965 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.sst_partitioner_factory: None 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55992b5bbc40) 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks: 1 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: pin_top_level_index_and_filter: 1 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_type: 0 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_index_type: 0 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_shortening: 1 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: checksum: 4 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: no_block_cache: 0 2024-06-01T02:45:14.966 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache: 0x55992b5bd090 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_name: BinnedLRUCache 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_options: 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: capacity : 536870912 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_shard_bits : 4 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: strict_capacity_limit : 0 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: high_pri_pool_ratio: 0.000 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_cache_compressed: (nil) 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: persistent_cache: (nil) 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size: 4096 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_size_deviation: 10 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_restart_interval: 16 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: index_block_restart_interval: 1 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: metadata_block_size: 4096 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: partition_filters: 0 2024-06-01T02:45:14.967 INFO:journalctl@ceph.mon.b.smithi146.stdout: use_delta_encoding: 1 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: filter_policy: bloomfilter 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: whole_key_filtering: 1 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: verify_compression: 0 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: read_amp_bytes_per_bit: 0 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: format_version: 5 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_index_compression: 1 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: block_align: 0 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: max_auto_readahead_size: 262144 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: prepopulate_block_cache: 0 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: initial_auto_readahead_size: 8192 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout: num_file_reads_for_auto_readahead: 2 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.write_buffer_size: 33554432 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_write_buffer_number: 2 2024-06-01T02:45:14.968 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression: NoCompression 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression: Disabled 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.prefix_extractor: nullptr 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.num_levels: 7 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-01T02:45:14.969 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.level: 32767 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.strategy: 0 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.enabled: false 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.target_file_size_base: 67108864 2024-06-01T02:45:14.970 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-06-01T02:45:14.971 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.arena_block_size: 1048576 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.disable_auto_compactions: 0 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-01T02:45:14.972 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.table_properties_collectors: 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.inplace_update_support: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.bloom_locality: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.max_successive_merges: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.paranoid_file_checks: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.force_consistency_checks: 1 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.report_bg_io_stats: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.ttl: 2592000 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-01T02:45:14.973 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.enable_blob_files: false 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.min_blob_size: 0 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.blob_file_size: 268435456 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.blob_file_starting_level: 0 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 27.sst 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-01T02:45:14.974 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 29, last_sequence is 14832, log_number is 25,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 25 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 39dce86c-dbd9-43b6-9941-49aae21c4be3 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209914577483, "job": 1, "event": "recovery_started", "wal_files": [25]} 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #25 mode 2 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209914580721, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 30, "file_size": 615856, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 14837, "largest_seqno": 15254, "table_properties": {"data_size": 613068, "index_size": 1262, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 517, "raw_key_size": 4824, "raw_average_key_size": 24, "raw_value_size": 608714, "raw_average_value_size": 3121, "num_data_blocks": 56, "num_entries": 195, "num_filter_entries": 195, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1717209914, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "39dce86c-dbd9-43b6-9941-49aae21c4be3", "db_session_id": "JWNOY5W18H88YJU7B7NG", "orig_file_number": 30, "seqno_to_time_mapping": "N/A"}} 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717209914580806, "job": 1, "event": "recovery_finished"} 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/version_set.cc:5047] Creating manifest 32 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000025.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55992b6ac000 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: DB pointer 0x55992b696000 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-06-01T02:45:14.975 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** DB Stats ** 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: L0 1/0 601.42 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 330.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: L6 1/0 11.24 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-06-01T02:45:14.976 INFO:journalctl@ceph.mon.b.smithi146.stdout: Sum 2/0 11.82 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 330.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 330.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** Compaction Stats [default] ** 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 330.0 0.00 0.00 1 0.002 0 0 0.0 0.0 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: Flush(GB): cumulative 0.001, interval 0.001 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-01T02:45:14.977 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: Cumulative compaction: 0.00 GB write, 107.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: Interval compaction: 0.00 GB write, 107.39 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache BinnedLRUCache@0x55992b5bd090#2 capacity: 512.00 MB usage: 1.92 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.59 KB,0.000113249%) IndexBlock(1,1.33 KB,0.00025332%) Misc(1,0.00 KB,0%) 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: starting mon.b rank 2 at public addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] at bind addrs [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???) e3 preinit fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).mds e1 new map 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).mds e1 print_map 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: e1 2024-06-01T02:45:14.978 INFO:journalctl@ceph.mon.b.smithi146.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout: legacy client fscid: -1 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout: 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout: No filesystems configured 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).osd e106 crush map has features 3314933000854323200, adjusting msgr requires 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).osd e106 crush map has features 432629239337189376, adjusting msgr requires 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 ceph-mon[57708]: mon.b@-1(???).paxosservice(auth 1..24) refresh upgraded, format 0 -> 3 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 bash[57690]: b8af55737ea256365dfbbb37fa45b92155a9266147813e3cf3c6f6167216baa9 2024-06-01T02:45:14.979 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:14 smithi146 systemd[1]: Started Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:45:15.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:45:14] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.33.4" 2024-06-01T02:45:18.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:18.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:20.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:20 smithi062 ceph-mon[64430]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:20.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:20 smithi062 ceph-mon[66884]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:21.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:21 smithi146 ceph-mon[57708]: pgmap v18: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: mon.b calling monitor election 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: mon.a calling monitor election 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: mon.c calling monitor election 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: monmap epoch 4 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: last_changed 2024-06-01T02:45:21.206403+0000 2024-06-01T02:45:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: min_mon_release 19 (squid) 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: election_strategy: 1 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: fsmap 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: mgrmap e40: y(active, since 38s), standbys: x 2024-06-01T02:45:22.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:22 smithi146 ceph-mon[57708]: overall HEALTH_OK 2024-06-01T02:45:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: mon.b calling monitor election 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: mon.a calling monitor election 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: mon.c calling monitor election 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: monmap epoch 4 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: last_changed 2024-06-01T02:45:21.206403+0000 2024-06-01T02:45:22.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: min_mon_release 19 (squid) 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: election_strategy: 1 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: fsmap 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: mgrmap e40: y(active, since 38s), standbys: x 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[64430]: overall HEALTH_OK 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: pgmap v22: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:22.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: mon.b calling monitor election 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: mon.a calling monitor election 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: mon.c calling monitor election 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: monmap epoch 4 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: last_changed 2024-06-01T02:45:21.206403+0000 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: created 2024-06-01T02:32:11.160426+0000 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: min_mon_release 19 (squid) 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: election_strategy: 1 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: 0: [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] mon.a 2024-06-01T02:45:22.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: 1: [v2:172.21.15.62:3301/0,v1:172.21.15.62:6790/0] mon.c 2024-06-01T02:45:22.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: 2: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-06-01T02:45:22.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: fsmap 2024-06-01T02:45:22.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: osdmap e106: 8 total, 8 up, 8 in 2024-06-01T02:45:22.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: mgrmap e40: y(active, since 38s), standbys: x 2024-06-01T02:45:22.519 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:22 smithi062 ceph-mon[66884]: overall HEALTH_OK 2024-06-01T02:45:23.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:23.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:24.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:24 smithi062 ceph-mon[64430]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:24.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:24 smithi062 ceph-mon[66884]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:24.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:24 smithi146 ceph-mon[57708]: pgmap v23: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:25.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:45:24.589+0000 7f15439f1700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-01T02:45:25.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:45:24] "GET /metrics HTTP/1.1" 200 38252 "" "Prometheus/2.33.4" 2024-06-01T02:45:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:25 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:25 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:25.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:25 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:25.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:25.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:26.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:26 smithi146 ceph-mon[57708]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:26.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:26 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:26.403 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:26 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:26 smithi062 ceph-mon[64430]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:26 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:26 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:26.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:26 smithi062 ceph-mon[66884]: pgmap v24: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:26.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:26 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:26.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:26 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:28.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:28 smithi146 ceph-mon[57708]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:28.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:28 smithi062 ceph-mon[66884]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:28.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:28 smithi062 ceph-mon[64430]: pgmap v25: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:28.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:28.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:28.660Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:29.832 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: Reconfiguring mon.a (monmap changed)... 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:29 smithi146 ceph-mon[57708]: Reconfiguring daemon mon.a on smithi062 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: Reconfiguring mon.a (monmap changed)... 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[64430]: Reconfiguring daemon mon.a on smithi062 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: pgmap v26: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: Reconfiguring mon.a (monmap changed)... 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:30.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:29 smithi062 ceph-mon[66884]: Reconfiguring daemon mon.a on smithi062 2024-06-01T02:45:30.529 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:45:30.529 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (2m) 26s ago 6m 14.3M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:45:30.529 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (113s) 6s ago 6m 45.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:45:30.529 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (83s) 26s ago 5m 51.5M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (73s) 6s ago 11m 450M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (107s) 26s ago 13m 494M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (58s) 26s ago 13m 41.9M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (16s) 6s ago 11m 17.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (32s) 26s ago 11m 17.1M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (2m) 26s ago 7m 17.9M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (117s) 6s ago 7m 14.9M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (11m) 26s ago 11m 53.4M 2736M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (10m) 26s ago 10m 56.0M 2736M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (10m) 26s ago 10m 51.3M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (9m) 26s ago 9m 53.4M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (9m) 6s ago 9m 56.7M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (9m) 6s ago 9m 53.8M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:45:30.530 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (8m) 6s ago 8m 52.8M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:45:30.531 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (8m) 6s ago 8m 56.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:45:30.531 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (78s) 6s ago 7m 64.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:45:30.531 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (6m) 26s ago 6m 75.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:45:30.531 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (6m) 6s ago 6m 75.0M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:45:30.531 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (5m) 26s ago 5m 75.2M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:45:30.531 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (5m) 6s ago 5m 74.0M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:45:30.955 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 5 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:45:30.956 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:45:31.159 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:31.159 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:31.159 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:31.159 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:45:31.160 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:30 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:31.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='client.34173 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:31.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='client.34179 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:31.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:31.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:31.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:45:31.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:45:31.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:30 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:45:31.319 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:45:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:45:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "5/23 daemons upgraded", 2024-06-01T02:45:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading mon daemons", 2024-06-01T02:45:31.320 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:45:31.320 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:45:31.741 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[66884]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[66884]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[66884]: Reconfiguring mgr.y (monmap changed)... 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[66884]: Reconfiguring daemon mgr.y on smithi062 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/955707927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/139101472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:45:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[64430]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:32.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[64430]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:32.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[64430]: Reconfiguring mgr.y (monmap changed)... 2024-06-01T02:45:32.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[64430]: Reconfiguring daemon mgr.y on smithi062 2024-06-01T02:45:32.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/955707927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:32.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:31 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/139101472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:45:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:31 smithi146 ceph-mon[57708]: pgmap v27: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:31 smithi146 ceph-mon[57708]: from='client.44113 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:31 smithi146 ceph-mon[57708]: Reconfiguring mgr.y (monmap changed)... 2024-06-01T02:45:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:31 smithi146 ceph-mon[57708]: Reconfiguring daemon mgr.y on smithi062 2024-06-01T02:45:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:31 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/955707927' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:31 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/139101472' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: Reconfiguring mon.c (monmap changed)... 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:33.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:33.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-mon[57708]: Reconfiguring daemon mon.c on smithi062 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: Reconfiguring mon.c (monmap changed)... 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[64430]: Reconfiguring daemon mon.c on smithi062 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: from='client.34197 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:33.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: Reconfiguring mon.c (monmap changed)... 2024-06-01T02:45:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:33 smithi062 ceph-mon[66884]: Reconfiguring daemon mon.c on smithi062 2024-06-01T02:45:33.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[64430]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[66884]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:45:34.435 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:34.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:34 smithi146 ceph-mon[57708]: pgmap v28: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:34.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:34 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:34.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:34 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:34.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:34 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:45:34.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:34 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:34.721 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:45:34] "GET /metrics HTTP/1.1" 200 38256 "" "Prometheus/2.33.4" 2024-06-01T02:45:35.341 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:35 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:35.341 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:35 smithi062 ceph-mon[64430]: Reconfiguring osd.0 (monmap changed)... 2024-06-01T02:45:35.341 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:35 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.0 on smithi062 2024-06-01T02:45:35.342 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:35 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:35.342 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:35 smithi062 ceph-mon[66884]: Reconfiguring osd.0 (monmap changed)... 2024-06-01T02:45:35.342 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:35 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.0 on smithi062 2024-06-01T02:45:35.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:35 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:35.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:35 smithi146 ceph-mon[57708]: Reconfiguring osd.0 (monmap changed)... 2024-06-01T02:45:35.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:35 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.0 on smithi062 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[64430]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[66884]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:36.369 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:36.370 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:36.370 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:45:36.370 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:36 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:36.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:36 smithi146 ceph-mon[57708]: pgmap v29: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:36.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:36 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:36.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:36 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:36.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:36 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:45:36.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:36 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:37 smithi146 ceph-mon[57708]: Reconfiguring osd.1 (monmap changed)... 2024-06-01T02:45:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:37 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.1 on smithi062 2024-06-01T02:45:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:45:37.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[64430]: Reconfiguring osd.1 (monmap changed)... 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.1 on smithi062 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[66884]: Reconfiguring osd.1 (monmap changed)... 2024-06-01T02:45:37.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.1 on smithi062 2024-06-01T02:45:37.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:37.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:37.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:45:37.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:38.413 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:38 smithi062 ceph-mon[64430]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:38.414 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:38 smithi062 ceph-mon[64430]: Reconfiguring osd.2 (monmap changed)... 2024-06-01T02:45:38.414 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:38 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.2 on smithi062 2024-06-01T02:45:38.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:38 smithi062 ceph-mon[66884]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:38.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:38 smithi062 ceph-mon[66884]: Reconfiguring osd.2 (monmap changed)... 2024-06-01T02:45:38.414 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:38 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.2 on smithi062 2024-06-01T02:45:38.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:38 smithi146 ceph-mon[57708]: pgmap v30: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:38.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:38 smithi146 ceph-mon[57708]: Reconfiguring osd.2 (monmap changed)... 2024-06-01T02:45:38.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:38 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.2 on smithi062 2024-06-01T02:45:38.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:38.658Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: Reconfiguring osd.3 (monmap changed)... 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:39 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.3 on smithi062 2024-06-01T02:45:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: Reconfiguring osd.3 (monmap changed)... 2024-06-01T02:45:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:45:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.3 on smithi062 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: pgmap v31: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: Reconfiguring osd.3 (monmap changed)... 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:39 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.3 on smithi062 2024-06-01T02:45:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: Reconfiguring rgw.foo.smithi062.lqlihd (monmap changed)... 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[64430]: Reconfiguring daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: Reconfiguring rgw.foo.smithi062.lqlihd (monmap changed)... 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:41 smithi062 ceph-mon[66884]: Reconfiguring daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: pgmap v32: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: Reconfiguring rgw.foo.smithi062.lqlihd (monmap changed)... 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:41.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:41 smithi146 ceph-mon[57708]: Reconfiguring daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: Reconfiguring rgw.smpl.smithi062.jsennv (monmap changed)... 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: Reconfiguring daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:45:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: Reconfiguring rgw.smpl.smithi062.jsennv (monmap changed)... 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: Reconfiguring daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:43.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:43.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: Reconfiguring rgw.smpl.smithi062.jsennv (monmap changed)... 2024-06-01T02:45:43.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:43.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:43.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: Reconfiguring daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:45:43.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:43.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:44.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:44 smithi146 ceph-mon[57708]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:44.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:44 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:44.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:44 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:44.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:44 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:44.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:44 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:44.441 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:44 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[64430]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[66884]: pgmap v33: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-01T02:45:44.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-01T02:45:44.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:45.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:45:44] "GET /metrics HTTP/1.1" 200 38261 "" "Prometheus/2.33.4" 2024-06-01T02:45:45.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:45 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:45.429 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:45 smithi146 ceph-mon[57708]: Reconfiguring mon.b (monmap changed)... 2024-06-01T02:45:45.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:45 smithi146 ceph-mon[57708]: Reconfiguring daemon mon.b on smithi146 2024-06-01T02:45:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:45 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:45 smithi062 ceph-mon[64430]: Reconfiguring mon.b (monmap changed)... 2024-06-01T02:45:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:45 smithi062 ceph-mon[64430]: Reconfiguring daemon mon.b on smithi146 2024-06-01T02:45:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:45 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:45 smithi062 ceph-mon[66884]: Reconfiguring mon.b (monmap changed)... 2024-06-01T02:45:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:45 smithi062 ceph-mon[66884]: Reconfiguring daemon mon.b on smithi146 2024-06-01T02:45:46.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:46 smithi146 ceph-mon[57708]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:46.433 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:46 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:46.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:46 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:46.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:46 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:45:46.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:46 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:45:46.434 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:46 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[64430]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:45:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:45:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[66884]: pgmap v34: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-01T02:45:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-01T02:45:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:46 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:47 smithi146 ceph-mon[57708]: Reconfiguring mgr.x (monmap changed)... 2024-06-01T02:45:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:47 smithi146 ceph-mon[57708]: Reconfiguring daemon mgr.x on smithi146 2024-06-01T02:45:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:47 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:47 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[64430]: Reconfiguring mgr.x (monmap changed)... 2024-06-01T02:45:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[64430]: Reconfiguring daemon mgr.x on smithi146 2024-06-01T02:45:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[66884]: Reconfiguring mgr.x (monmap changed)... 2024-06-01T02:45:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[66884]: Reconfiguring daemon mgr.x on smithi146 2024-06-01T02:45:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:47 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:48 smithi146 ceph-mon[57708]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:48 smithi146 ceph-mon[57708]: Reconfiguring osd.4 (monmap changed)... 2024-06-01T02:45:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:48 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:45:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:48 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:48 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.4 on smithi146 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[64430]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[64430]: Reconfiguring osd.4 (monmap changed)... 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.4 on smithi146 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[66884]: pgmap v35: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[66884]: Reconfiguring osd.4 (monmap changed)... 2024-06-01T02:45:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:45:48.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:48.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:48 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.4 on smithi146 2024-06-01T02:45:48.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:49.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:49.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:49.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:49.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: Reconfiguring osd.5 (monmap changed)... 2024-06-01T02:45:49.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:45:49.908 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:49.909 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:49 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.5 on smithi146 2024-06-01T02:45:50.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:50.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:50.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:50.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: Reconfiguring osd.5 (monmap changed)... 2024-06-01T02:45:50.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:45:50.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.5 on smithi146 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: pgmap v36: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: Reconfiguring osd.5 (monmap changed)... 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:50.249 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:49 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.5 on smithi146 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: Reconfiguring osd.6 (monmap changed)... 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.6 on smithi146 2024-06-01T02:45:51.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:51 smithi146 ceph-mon[57708]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: Reconfiguring osd.6 (monmap changed)... 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.6 on smithi146 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[66884]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:51.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:51.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:51.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: Reconfiguring osd.6 (monmap changed)... 2024-06-01T02:45:51.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:45:51.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:51.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.6 on smithi146 2024-06-01T02:45:51.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:51 smithi062 ceph-mon[64430]: pgmap v37: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:53.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:53.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:53.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:52 smithi146 ceph-mon[57708]: Reconfiguring osd.7 (monmap changed)... 2024-06-01T02:45:53.204 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:45:53.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:53.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:52 smithi146 ceph-mon[57708]: Reconfiguring daemon osd.7 on smithi146 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[66884]: Reconfiguring osd.7 (monmap changed)... 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[66884]: Reconfiguring daemon osd.7 on smithi146 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:53.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[64430]: Reconfiguring osd.7 (monmap changed)... 2024-06-01T02:45:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:45:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:52 smithi062 ceph-mon[64430]: Reconfiguring daemon osd.7 on smithi146 2024-06-01T02:45:53.800 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:53.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:54.166 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:53 smithi146 ceph-mon[57708]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:54.167 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[66884]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[64430]: pgmap v38: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:54.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:54.898 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:45:54] "GET /metrics HTTP/1.1" 200 38261 "" "Prometheus/2.33.4" 2024-06-01T02:45:55.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:54 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:55.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:54 smithi146 ceph-mon[57708]: Reconfiguring rgw.foo.smithi146.btivlw (monmap changed)... 2024-06-01T02:45:55.161 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:54 smithi146 ceph-mon[57708]: Reconfiguring daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:45:55.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:55.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-mon[64430]: Reconfiguring rgw.foo.smithi146.btivlw (monmap changed)... 2024-06-01T02:45:55.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-mon[64430]: Reconfiguring daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:45:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:45:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-mon[66884]: Reconfiguring rgw.foo.smithi146.btivlw (monmap changed)... 2024-06-01T02:45:55.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:54 smithi062 ceph-mon[66884]: Reconfiguring daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:45:56.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:56 smithi146 ceph-mon[57708]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:56.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:56 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:56.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:56 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:56.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:56 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:56.452 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:56 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[66884]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[64430]: pgmap v39: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:56.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:56.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:45:56.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:56 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: Reconfiguring rgw.smpl.smithi146.olokyu (monmap changed)... 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: Reconfiguring daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: Reconfiguring rgw.smpl.smithi146.olokyu (monmap changed)... 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: Reconfiguring daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:45:57.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-01T02:45:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: Reconfiguring rgw.smpl.smithi146.olokyu (monmap changed)... 2024-06-01T02:45:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: Reconfiguring daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:45:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.a"}]': finished 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.b"}]': finished 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.c"}]': finished 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-01T02:45:58.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[64430]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:58.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all mon 2024-06-01T02:45:58.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all crash 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[64430]: Upgrade: osd.0 is safe to restart 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[66884]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all mon 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all crash 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[66884]: Upgrade: osd.0 is safe to restart 2024-06-01T02:45:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:58.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-mon[57708]: pgmap v40: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:45:58.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all mon 2024-06-01T02:45:58.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all crash 2024-06-01T02:45:58.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-06-01T02:45:58.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-mon[57708]: Upgrade: osd.0 is safe to restart 2024-06-01T02:45:58.660 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:45:58.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:45:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:45:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:45:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:59 smithi146 ceph-mon[57708]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:45:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:59 smithi146 ceph-mon[57708]: Upgrade: Updating osd.0 2024-06-01T02:45:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:45:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:45:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:45:59.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:45:59 smithi146 ceph-mon[57708]: Deploying daemon osd.0 on smithi062 2024-06-01T02:46:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[64430]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:46:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[64430]: Upgrade: Updating osd.0 2024-06-01T02:46:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:46:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:00.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[64430]: Deploying daemon osd.0 on smithi062 2024-06-01T02:46:00.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[66884]: pgmap v41: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:46:00.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[66884]: Upgrade: Updating osd.0 2024-06-01T02:46:00.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:00.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-01T02:46:00.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:00.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:45:59 smithi062 ceph-mon[66884]: Deploying daemon osd.0 on smithi062 2024-06-01T02:46:01.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:01 smithi146 ceph-mon[57708]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:02.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:01 smithi062 ceph-mon[64430]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:02.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:01 smithi062 ceph-mon[66884]: pgmap v42: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:02.134 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:46:02.367 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:02 smithi062 systemd[1]: Stopping Ceph osd.0 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:46:02.698 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[32673]: 2024-06-01T02:46:02.366+0000 7f5301770700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:46:02.698 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[32673]: 2024-06-01T02:46:02.366+0000 7f5301770700 -1 osd.0 106 *** Got signal Terminated *** 2024-06-01T02:46:02.699 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[32673]: 2024-06-01T02:46:02.366+0000 7f5301770700 -1 osd.0 106 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:46:02.854 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (2m) 58s ago 7m 14.3M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (2m) 38s ago 7m 45.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (115s) 58s ago 6m 51.5M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (106s) 38s ago 11m 450M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (2m) 58s ago 13m 494M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (91s) 58s ago 13m 41.9M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (48s) 38s ago 12m 17.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (64s) 58s ago 12m 17.1M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (2m) 58s ago 8m 17.9M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (2m) 38s ago 8m 14.9M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (11m) 58s ago 11m 53.4M 2736M 17.2.0 e1d6a67b021e 34a94ae44cbe 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (11m) 58s ago 11m 56.0M 2736M 17.2.0 e1d6a67b021e cb5ac4403edb 2024-06-01T02:46:02.855 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (10m) 58s ago 10m 51.3M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (10m) 58s ago 10m 53.4M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (10m) 38s ago 10m 56.7M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (9m) 38s ago 9m 53.8M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (9m) 38s ago 9m 52.8M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (8m) 38s ago 8m 56.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (111s) 38s ago 8m 64.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (6m) 58s ago 6m 75.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (6m) 38s ago 6m 75.0M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (6m) 58s ago 6m 75.2M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:46:02.856 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (6m) 38s ago 6m 74.0M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:46:02.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:02 smithi146 ceph-mon[57708]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:02.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:02 smithi146 ceph-mon[57708]: osd.0 marked itself down and dead 2024-06-01T02:46:03.012 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-mon[64430]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:03.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-mon[64430]: osd.0 marked itself down and dead 2024-06-01T02:46:03.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-mon[66884]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:03.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:02 smithi062 ceph-mon[66884]: osd.0 marked itself down and dead 2024-06-01T02:46:03.283 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:46:03.284 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:03.285 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:46:03.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 11, 2024-06-01T02:46:03.285 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 5 2024-06-01T02:46:03.285 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:46:03.285 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:46:03.334 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:03 smithi062 podman[69739]: 2024-06-01 02:46:03.048379077 +0000 UTC m=+0.783668156 container died 34a94ae44cbe28299f275d1133a14c6f12c47606224f22a8c059e93f13cb6cdb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, distribution-scope=public, name=centos-stream, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-06-01T02:46:03.644 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:46:03.644 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:46:03.644 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:46:03.644 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "5/23 daemons upgraded", 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:46:03.645 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:46:03.645 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:03 smithi062 podman[69739]: 2024-06-01 02:46:03.425500764 +0000 UTC m=+1.160789841 container cleanup 34a94ae44cbe28299f275d1133a14c6f12c47606224f22a8c059e93f13cb6cdb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , version=8, release=754, vcs-type=git, distribution-scope=public) 2024-06-01T02:46:03.645 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:03 smithi062 bash[69739]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0 2024-06-01T02:46:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-mon[57708]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-mon[57708]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-mon[57708]: osdmap e107: 8 total, 7 up, 8 in 2024-06-01T02:46:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-mon[57708]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/417019867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:03.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:03.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:04.013 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:03 smithi062 podman[69801]: 2024-06-01 02:46:03.752520909 +0000 UTC m=+0.697204532 container remove 34a94ae44cbe28299f275d1133a14c6f12c47606224f22a8c059e93f13cb6cdb (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, version=8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, ceph=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, RELEASE=HEAD, vendor=Red Hat, Inc.) 2024-06-01T02:46:04.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[64430]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:04.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[64430]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:04.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:04.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[64430]: osdmap e107: 8 total, 7 up, 8 in 2024-06-01T02:46:04.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[64430]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:04.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/417019867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[66884]: from='client.34215 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[66884]: pgmap v43: 161 pgs: 161 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[66884]: osdmap e107: 8 total, 7 up, 8 in 2024-06-01T02:46:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[66884]: from='client.44149 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:03 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/417019867' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:04.084 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 osds down 2024-06-01T02:46:04.084 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] OSD_DOWN: 1 osds down 2024-06-01T02:46:04.085 INFO:teuthology.orchestra.run.smithi062.stdout: osd.0 (root=default,host=smithi062) is down 2024-06-01T02:46:04.405 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:04 smithi062 podman[69933]: 2024-06-01 02:46:04.007468561 +0000 UTC m=+0.018299514 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:04.405 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:04 smithi062 podman[69933]: 2024-06-01 02:46:04.111829711 +0000 UTC m=+0.122660649 container create 0efb8d123cf578798342a11a013f193c44b87e200327291ac3bb5874da8d6dfb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:46:04.405 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:04 smithi062 podman[69933]: 2024-06-01 02:46:04.322694981 +0000 UTC m=+0.333525920 container init 0efb8d123cf578798342a11a013f193c44b87e200327291ac3bb5874da8d6dfb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-01T02:46:04.406 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:04 smithi062 podman[69933]: 2024-06-01 02:46:04.326219146 +0000 UTC m=+0.337050085 container start 0efb8d123cf578798342a11a013f193c44b87e200327291ac3bb5874da8d6dfb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True) 2024-06-01T02:46:04.406 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:04 smithi062 podman[69933]: 2024-06-01 02:46:04.405670789 +0000 UTC m=+0.416501748 container attach 0efb8d123cf578798342a11a013f193c44b87e200327291ac3bb5874da8d6dfb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True) 2024-06-01T02:46:04.712 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:46:04] "GET /metrics HTTP/1.1" 200 38195 "" "Prometheus/2.33.4" 2024-06-01T02:46:04.712 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:04 smithi062 podman[69933]: 2024-06-01 02:46:04.628833034 +0000 UTC m=+0.639663973 container died 0efb8d123cf578798342a11a013f193c44b87e200327291ac3bb5874da8d6dfb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc) 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[64430]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[64430]: osdmap e108: 8 total, 7 up, 8 in 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1934969025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[66884]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[66884]: osdmap e108: 8 total, 7 up, 8 in 2024-06-01T02:46:05.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:04 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1934969025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:46:05.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:04 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:05.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:04 smithi146 ceph-mon[57708]: from='client.34233 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:05.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:04 smithi146 ceph-mon[57708]: osdmap e108: 8 total, 7 up, 8 in 2024-06-01T02:46:05.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:04 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1934969025' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:46:05.395 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 podman[69962]: 2024-06-01 02:46:05.159297724 +0000 UTC m=+0.524257596 container remove 0efb8d123cf578798342a11a013f193c44b87e200327291ac3bb5874da8d6dfb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-01T02:46:05.395 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-01T02:46:05.395 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 systemd[1]: Stopped Ceph osd.0 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:46:05.395 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0.service: Consumed 7.739s CPU time. 2024-06-01T02:46:05.713 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 systemd[1]: Starting Ceph osd.0 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:46:05.713 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 podman[70059]: 2024-06-01 02:46:05.519508808 +0000 UTC m=+0.018720646 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:05.713 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 podman[70059]: 2024-06-01 02:46:05.635674335 +0000 UTC m=+0.134886168 container create 1a3332880eff8be444a0418ff1be3ca0623c734891f8693eaa3d41f7bdf85496 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-01T02:46:05.971 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:05 smithi062 ceph-mon[64430]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:05.971 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:05 smithi062 ceph-mon[64430]: Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-06-01T02:46:05.971 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:05 smithi062 ceph-mon[66884]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:05.971 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:05 smithi062 ceph-mon[66884]: Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-06-01T02:46:05.972 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 podman[70059]: 2024-06-01 02:46:05.889093348 +0000 UTC m=+0.388305182 container init 1a3332880eff8be444a0418ff1be3ca0623c734891f8693eaa3d41f7bdf85496 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-01T02:46:05.972 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 podman[70059]: 2024-06-01 02:46:05.892972524 +0000 UTC m=+0.392184367 container start 1a3332880eff8be444a0418ff1be3ca0623c734891f8693eaa3d41f7bdf85496 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-06-01T02:46:05.972 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:05 smithi062 podman[70059]: 2024-06-01 02:46:05.971532371 +0000 UTC m=+0.470744232 container attach 1a3332880eff8be444a0418ff1be3ca0623c734891f8693eaa3d41f7bdf85496 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:46:06.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:05 smithi146 ceph-mon[57708]: pgmap v46: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:06.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:05 smithi146 ceph-mon[57708]: Health check failed: Reduced data availability: 13 pgs inactive, 57 pgs peering (PG_AVAILABILITY) 2024-06-01T02:46:06.666 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-01T02:46:06.666 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 bash[70059]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate[70073]: --> ceph-volume raw activate successful for osd ID: 0 2024-06-01T02:46:06.667 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 podman[70059]: 2024-06-01 02:46:06.385035871 +0000 UTC m=+0.884247704 container died 1a3332880eff8be444a0418ff1be3ca0623c734891f8693eaa3d41f7bdf85496 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0) 2024-06-01T02:46:07.238 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:06 smithi062 podman[70209]: 2024-06-01 02:46:06.941970903 +0000 UTC m=+0.550898030 container remove 1a3332880eff8be444a0418ff1be3ca0623c734891f8693eaa3d41f7bdf85496 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-01T02:46:07.238 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 podman[70268]: 2024-06-01 02:46:07.15105046 +0000 UTC m=+0.109325587 container create fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-01T02:46:07.238 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 podman[70268]: 2024-06-01 02:46:07.059956733 +0000 UTC m=+0.018231885 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:07.513 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 podman[70268]: 2024-06-01 02:46:07.337176116 +0000 UTC m=+0.295451253 container init fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:46:07.513 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 podman[70268]: 2024-06-01 02:46:07.340540092 +0000 UTC m=+0.298815231 container start fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux ) 2024-06-01T02:46:07.513 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-osd[70287]: -- 172.21.15.62:0/3986166303 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560a4b94f680 con 0x560a4b919400 2024-06-01T02:46:07.513 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 bash[70268]: fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb 2024-06-01T02:46:07.513 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 systemd[1]: Started Ceph osd.0 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:46:08.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-mon[64430]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:08.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:08.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:08.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-mon[66884]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:08.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:08.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:08.014 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:07 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:46:07.887+0000 7fc4032c77c0 -1 Falling back to public interface 2024-06-01T02:46:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:07 smithi146 ceph-mon[57708]: pgmap v47: 161 pgs: 58 peering, 103 active+clean; 585 KiB data, 98 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:08.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:08 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 77/723 objects degraded (10.650%), 19 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:08.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:08 smithi146 ceph-mon[57708]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2024-06-01T02:46:08.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:09.023 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:08 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 77/723 objects degraded (10.650%), 19 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:09.023 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:08 smithi062 ceph-mon[64430]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2024-06-01T02:46:09.024 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:08 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 77/723 objects degraded (10.650%), 19 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:09.024 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:08 smithi062 ceph-mon[66884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 57 pgs peering) 2024-06-01T02:46:10.087 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:09 smithi062 ceph-mon[64430]: pgmap v48: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-01T02:46:10.088 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:09 smithi062 ceph-mon[66884]: pgmap v48: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-01T02:46:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:09 smithi146 ceph-mon[57708]: pgmap v48: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-01T02:46:11.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:10 smithi062 ceph-mon[66884]: from='osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:46:11.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:10 smithi062 ceph-mon[66884]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:46:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:10 smithi062 ceph-mon[64430]: from='osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:46:11.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:10 smithi062 ceph-mon[64430]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:46:11.013 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:46:10.618+0000 7fc4032c77c0 -1 osd.0 0 read_superblock omap replica is missing. 2024-06-01T02:46:11.013 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:10 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:46:10.659+0000 7fc4032c77c0 -1 osd.0 106 log_to_monitors true 2024-06-01T02:46:11.205 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:10 smithi146 ceph-mon[57708]: from='osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:46:11.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:10 smithi146 ceph-mon[57708]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-01T02:46:11.511 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:46:11.237+0000 7fc3f0716700 -1 osd.0 106 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:46:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:11 smithi146 ceph-mon[57708]: pgmap v49: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-01T02:46:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:11 smithi146 ceph-mon[57708]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T02:46:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:11 smithi146 ceph-mon[57708]: osdmap e109: 8 total, 7 up, 8 in 2024-06-01T02:46:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:11 smithi146 ceph-mon[57708]: from='osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:11 smithi146 ceph-mon[57708]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:11 smithi146 ceph-mon[57708]: from='osd.0 ' entity='osd.0' 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[64430]: pgmap v49: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[64430]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[64430]: osdmap e109: 8 total, 7 up, 8 in 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[64430]: from='osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[64430]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[64430]: from='osd.0 ' entity='osd.0' 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[66884]: pgmap v49: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 99 MiB used, 715 GiB / 715 GiB avail; 77/723 objects degraded (10.650%) 2024-06-01T02:46:12.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[66884]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-01T02:46:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[66884]: osdmap e109: 8 total, 7 up, 8 in 2024-06-01T02:46:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[66884]: from='osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[66884]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:12.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:11 smithi062 ceph-mon[66884]: from='osd.0 ' entity='osd.0' 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115] boot 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: osdmap e110: 8 total, 8 up, 8 in 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:46:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115] boot 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: osdmap e110: 8 total, 8 up, 8 in 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: osd.0 [v2:172.21.15.62:6802/2984231115,v1:172.21.15.62:6803/2984231115] boot 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: osdmap e110: 8 total, 8 up, 8 in 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:13.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:13.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:14.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[64430]: pgmap v52: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/723 objects degraded (10.650%) 2024-06-01T02:46:14.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:14.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:14.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[64430]: osdmap e111: 8 total, 8 up, 8 in 2024-06-01T02:46:14.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[66884]: pgmap v52: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/723 objects degraded (10.650%) 2024-06-01T02:46:14.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:14.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:14.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:13 smithi062 ceph-mon[66884]: osdmap e111: 8 total, 8 up, 8 in 2024-06-01T02:46:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:13 smithi146 ceph-mon[57708]: pgmap v52: 161 pgs: 19 active+undersized+degraded, 39 active+undersized, 103 active+clean; 585 KiB data, 114 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 77/723 objects degraded (10.650%) 2024-06-01T02:46:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:13 smithi146 ceph-mon[57708]: osdmap e111: 8 total, 8 up, 8 in 2024-06-01T02:46:14.912 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:46:14] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.33.4" 2024-06-01T02:46:15.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:14 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:15.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:14 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:15.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:14 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:16.129 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:15 smithi062 ceph-mon[64430]: pgmap v54: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-06-01T02:46:16.129 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:15 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:16.129 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:15 smithi062 ceph-mon[66884]: pgmap v54: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-06-01T02:46:16.129 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:15 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:15 smithi146 ceph-mon[57708]: pgmap v54: 161 pgs: 8 active+undersized+degraded, 13 active+undersized, 140 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 33/723 objects degraded (4.564%) 2024-06-01T02:46:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:15 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:17.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-01T02:46:17.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[66884]: Upgrade: osd.1 is safe to restart 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-01T02:46:17.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-01T02:46:17.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:17 smithi062 ceph-mon[64430]: Upgrade: osd.1 is safe to restart 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: pgmap v55: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:17.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:17.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-01T02:46:17.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-06-01T02:46:17.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:17 smithi146 ceph-mon[57708]: Upgrade: osd.1 is safe to restart 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded) 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: Upgrade: Updating osd.1 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[64430]: Deploying daemon osd.1 on smithi062 2024-06-01T02:46:18.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded) 2024-06-01T02:46:18.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:46:18.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: Upgrade: Updating osd.1 2024-06-01T02:46:18.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:18.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:46:18.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:18.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:18 smithi062 ceph-mon[66884]: Deploying daemon osd.1 on smithi062 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 33/723 objects degraded (4.564%), 8 pgs degraded) 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: Upgrade: Updating osd.1 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:18.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-mon[57708]: Deploying daemon osd.1 on smithi062 2024-06-01T02:46:18.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:19.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:19 smithi062 ceph-mon[64430]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:46:19.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:19 smithi062 ceph-mon[66884]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:46:19.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:19 smithi146 ceph-mon[57708]: pgmap v56: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:46:21.513 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:21 smithi062 systemd[1]: Stopping Ceph osd.1 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:46:21.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:21 smithi146 ceph-mon[57708]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:46:22.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:21 smithi062 ceph-mon[64430]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:46:22.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:21 smithi062 ceph-mon[66884]: pgmap v57: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:46:22.013 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[35614]: 2024-06-01T02:46:21.591+0000 7f3115c1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:46:22.013 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[35614]: 2024-06-01T02:46:21.591+0000 7f3115c1c700 -1 osd.1 111 *** Got signal Terminated *** 2024-06-01T02:46:22.014 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[35614]: 2024-06-01T02:46:21.591+0000 7f3115c1c700 -1 osd.1 111 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:46:22.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:22 smithi146 ceph-mon[57708]: osd.1 marked itself down and dead 2024-06-01T02:46:22.990 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:22 smithi062 ceph-mon[64430]: osd.1 marked itself down and dead 2024-06-01T02:46:22.991 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:22 smithi062 ceph-mon[66884]: osd.1 marked itself down and dead 2024-06-01T02:46:23.250 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:22 smithi062 podman[72108]: 2024-06-01 02:46:22.991207849 +0000 UTC m=+1.511177117 container died cb5ac4403edba296b0656552de6c62fb14b25eebd9596ab90223fec88992c2ca (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, architecture=x86_64, release=754, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-06-01T02:46:23.513 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:23 smithi062 podman[72108]: 2024-06-01 02:46:23.341218818 +0000 UTC m=+1.861188078 container cleanup cb5ac4403edba296b0656552de6c62fb14b25eebd9596ab90223fec88992c2ca (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, maintainer=Guillaume Abrioux , architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-06-01T02:46:23.513 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:23 smithi062 bash[72108]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1 2024-06-01T02:46:23.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:23 smithi146 ceph-mon[57708]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:23.871 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:23 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:23.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:23 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:23.872 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:23 smithi146 ceph-mon[57708]: osdmap e112: 8 total, 7 up, 8 in 2024-06-01T02:46:23.872 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:23.506Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[64430]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[64430]: osdmap e112: 8 total, 7 up, 8 in 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[66884]: pgmap v58: 161 pgs: 161 active+clean; 585 KiB data, 115 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:23.985 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:23.986 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:23 smithi062 ceph-mon[66884]: osdmap e112: 8 total, 7 up, 8 in 2024-06-01T02:46:23.986 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:23 smithi062 podman[72123]: 2024-06-01 02:46:23.517264771 +0000 UTC m=+0.520813825 container remove cb5ac4403edba296b0656552de6c62fb14b25eebd9596ab90223fec88992c2ca (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, version=8, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , release=754, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, ceph=True, name=centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc.) 2024-06-01T02:46:24.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:23 smithi062 podman[72213]: 2024-06-01 02:46:23.985058479 +0000 UTC m=+0.104960442 container create 423b289be4d77b4a26491bd693349088093e4ade4f5455da27986fe7290a30d4 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc) 2024-06-01T02:46:24.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:23 smithi062 podman[72213]: 2024-06-01 02:46:23.898400194 +0000 UTC m=+0.018302163 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:24.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:24 smithi062 podman[72213]: 2024-06-01 02:46:24.196183677 +0000 UTC m=+0.316085653 container init 423b289be4d77b4a26491bd693349088093e4ade4f5455da27986fe7290a30d4 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux ) 2024-06-01T02:46:24.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:24 smithi062 podman[72213]: 2024-06-01 02:46:24.19963867 +0000 UTC m=+0.319540642 container start 423b289be4d77b4a26491bd693349088093e4ade4f5455da27986fe7290a30d4 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-01T02:46:24.585 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:24 smithi062 podman[72213]: 2024-06-01 02:46:24.296156846 +0000 UTC m=+0.416058823 container attach 423b289be4d77b4a26491bd693349088093e4ade4f5455da27986fe7290a30d4 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, RELEASE=squid-3d89acc, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-01T02:46:24.585 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:24 smithi062 podman[72213]: 2024-06-01 02:46:24.512349602 +0000 UTC m=+0.632251571 container died 423b289be4d77b4a26491bd693349088093e4ade4f5455da27986fe7290a30d4 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:46:24.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:24 smithi146 ceph-mon[57708]: osdmap e113: 8 total, 7 up, 8 in 2024-06-01T02:46:24.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:24 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:25.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:24 smithi062 ceph-mon[64430]: osdmap e113: 8 total, 7 up, 8 in 2024-06-01T02:46:25.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:24 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:25.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:46:24] "GET /metrics HTTP/1.1" 200 38194 "" "Prometheus/2.33.4" 2024-06-01T02:46:25.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:24 smithi062 ceph-mon[66884]: osdmap e113: 8 total, 7 up, 8 in 2024-06-01T02:46:25.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:24 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:25.501 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 podman[72239]: 2024-06-01 02:46:25.058400768 +0000 UTC m=+0.539574900 container remove 423b289be4d77b4a26491bd693349088093e4ade4f5455da27986fe7290a30d4 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:46:25.501 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.1.service: Deactivated successfully. 2024-06-01T02:46:25.501 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 systemd[1]: Stopped Ceph osd.1 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:46:25.501 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.1.service: Consumed 8.665s CPU time. 2024-06-01T02:46:25.501 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 systemd[1]: Starting Ceph osd.1 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:46:25.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:25 smithi062 ceph-mon[64430]: pgmap v61: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:25.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:25 smithi062 ceph-mon[64430]: Health check failed: Reduced data availability: 11 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-06-01T02:46:25.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:25 smithi062 ceph-mon[66884]: pgmap v61: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:25.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:25 smithi062 ceph-mon[66884]: Health check failed: Reduced data availability: 11 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-06-01T02:46:25.764 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 podman[72338]: 2024-06-01 02:46:25.501251531 +0000 UTC m=+0.114182764 container create c701b88751b74360042bd534fb4f6ee728a53e887da1ee82aa2faedb9fda9316 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-01T02:46:25.764 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 podman[72338]: 2024-06-01 02:46:25.405061849 +0000 UTC m=+0.017993080 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:25.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:25 smithi146 ceph-mon[57708]: pgmap v61: 161 pgs: 56 peering, 1 stale+active+clean, 104 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:46:25.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:25 smithi146 ceph-mon[57708]: Health check failed: Reduced data availability: 11 pgs inactive, 16 pgs peering (PG_AVAILABILITY) 2024-06-01T02:46:26.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 podman[72338]: 2024-06-01 02:46:25.879245549 +0000 UTC m=+0.492176782 container init c701b88751b74360042bd534fb4f6ee728a53e887da1ee82aa2faedb9fda9316 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-06-01T02:46:26.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 podman[72338]: 2024-06-01 02:46:25.882847101 +0000 UTC m=+0.495778334 container start c701b88751b74360042bd534fb4f6ee728a53e887da1ee82aa2faedb9fda9316 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-01T02:46:26.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:25 smithi062 podman[72338]: 2024-06-01 02:46:25.954326083 +0000 UTC m=+0.567257334 container attach c701b88751b74360042bd534fb4f6ee728a53e887da1ee82aa2faedb9fda9316 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 bash[72338]: --> ceph-volume raw activate successful for osd ID: 1 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-01T02:46:26.639 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-06-01T02:46:26.640 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-01T02:46:26.640 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate[72352]: --> ceph-volume raw activate successful for osd ID: 1 2024-06-01T02:46:26.640 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 podman[72338]: 2024-06-01 02:46:26.374369221 +0000 UTC m=+0.987300452 container died c701b88751b74360042bd534fb4f6ee728a53e887da1ee82aa2faedb9fda9316 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-01T02:46:26.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:26 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-06-01T02:46:26.957 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-06-01T02:46:26.957 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:26 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 6/723 objects degraded (0.830%), 1 pg degraded (PG_DEGRADED) 2024-06-01T02:46:27.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:26 smithi062 podman[72473]: 2024-06-01 02:46:26.958089451 +0000 UTC m=+0.577553811 container remove c701b88751b74360042bd534fb4f6ee728a53e887da1ee82aa2faedb9fda9316 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-06-01T02:46:27.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 podman[72532]: 2024-06-01 02:46:27.167370407 +0000 UTC m=+0.110316302 container create 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc) 2024-06-01T02:46:27.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 podman[72532]: 2024-06-01 02:46:27.074975562 +0000 UTC m=+0.017921401 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:27.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 podman[72532]: 2024-06-01 02:46:27.370419237 +0000 UTC m=+0.313365087 container init 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-01T02:46:27.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 podman[72532]: 2024-06-01 02:46:27.373803177 +0000 UTC m=+0.316749017 container start 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-01T02:46:27.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-osd[72550]: -- 172.21.15.62:0/1735376474 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e6e6967680 con 0x55e6e6930c00 2024-06-01T02:46:27.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 bash[72532]: 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc 2024-06-01T02:46:27.642 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 systemd[1]: Started Ceph osd.1 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:46:27.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[64430]: pgmap v62: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-06-01T02:46:27.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[66884]: pgmap v62: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:27.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:27 smithi146 ceph-mon[57708]: pgmap v62: 161 pgs: 1 active+undersized+degraded, 56 peering, 6 active+undersized, 98 active+clean; 585 KiB data, 116 MiB used, 715 GiB / 715 GiB avail; 6/723 objects degraded (0.830%) 2024-06-01T02:46:27.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:27.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:28.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:27 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:46:27.918+0000 7f4be60b77c0 -1 Falling back to public interface 2024-06-01T02:46:28.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:28 smithi146 ceph-mon[57708]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 16 pgs peering) 2024-06-01T02:46:28.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:29.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:28 smithi062 ceph-mon[64430]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 16 pgs peering) 2024-06-01T02:46:29.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:28 smithi062 ceph-mon[66884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 11 pgs inactive, 16 pgs peering) 2024-06-01T02:46:29.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:29 smithi146 ceph-mon[57708]: pgmap v63: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 98/723 objects degraded (13.555%) 2024-06-01T02:46:30.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:29 smithi062 ceph-mon[64430]: pgmap v63: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 98/723 objects degraded (13.555%) 2024-06-01T02:46:30.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:29 smithi062 ceph-mon[66884]: pgmap v63: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 98/723 objects degraded (13.555%) 2024-06-01T02:46:30.930 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:30 smithi062 ceph-mon[64430]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:46:30.930 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:46:30.647+0000 7f4be60b77c0 -1 osd.1 0 read_superblock omap replica is missing. 2024-06-01T02:46:30.930 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:46:30.678+0000 7f4be60b77c0 -1 osd.1 111 log_to_monitors true 2024-06-01T02:46:30.931 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:46:30 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:46:30.726+0000 7f4bd750e700 -1 osd.1 111 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:46:30.931 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:30 smithi062 ceph-mon[66884]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:46:30.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:30 smithi146 ceph-mon[57708]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-01T02:46:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[64430]: pgmap v64: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 98/723 objects degraded (13.555%) 2024-06-01T02:46:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[64430]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T02:46:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[64430]: osdmap e114: 8 total, 7 up, 8 in 2024-06-01T02:46:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[64430]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[66884]: pgmap v64: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 98/723 objects degraded (13.555%) 2024-06-01T02:46:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[66884]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T02:46:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[66884]: osdmap e114: 8 total, 7 up, 8 in 2024-06-01T02:46:32.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:31 smithi062 ceph-mon[66884]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:31 smithi146 ceph-mon[57708]: pgmap v64: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 120 MiB used, 715 GiB / 715 GiB avail; 98/723 objects degraded (13.555%) 2024-06-01T02:46:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:31 smithi146 ceph-mon[57708]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-01T02:46:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:31 smithi146 ceph-mon[57708]: osdmap e114: 8 total, 7 up, 8 in 2024-06-01T02:46:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:31 smithi146 ceph-mon[57708]: from='osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342] boot 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: osdmap e115: 8 total, 8 up, 8 in 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 98/723 objects degraded (13.555%), 20 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:33.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342] boot 2024-06-01T02:46:33.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: osdmap e115: 8 total, 8 up, 8 in 2024-06-01T02:46:33.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:46:33.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 98/723 objects degraded (13.555%), 20 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:33.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: osd.1 [v2:172.21.15.62:6810/544378342,v1:172.21.15.62:6811/544378342] boot 2024-06-01T02:46:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: osdmap e115: 8 total, 8 up, 8 in 2024-06-01T02:46:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-01T02:46:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 98/723 objects degraded (13.555%), 20 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[64430]: pgmap v67: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 98/723 objects degraded (13.555%) 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[64430]: osdmap e116: 8 total, 8 up, 8 in 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[66884]: pgmap v67: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 98/723 objects degraded (13.555%) 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[66884]: osdmap e116: 8 total, 8 up, 8 in 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.932 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:33 smithi146 ceph-mon[57708]: pgmap v67: 161 pgs: 20 active+undersized+degraded, 43 active+undersized, 98 active+clean; 585 KiB data, 135 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 98/723 objects degraded (13.555%) 2024-06-01T02:46:33.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:33 smithi146 ceph-mon[57708]: osdmap e116: 8 total, 8 up, 8 in 2024-06-01T02:46:33.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:33.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:34.451 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:46:34.951 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:34 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:34.951 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:34 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:34.952 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:46:34] "GET /metrics HTTP/1.1" 200 38405 "" "Prometheus/2.33.4" 2024-06-01T02:46:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:34 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:35.140 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:46:35.140 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (3m) 2s ago 7m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:46:35.140 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (2m) 70s ago 7m 45.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (2m) 2s ago 6m 53.8M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (2m) 70s ago 12m 450M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (2m) 2s ago 14m 503M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (2m) 2s ago 14m 47.1M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (80s) 70s ago 12m 17.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (97s) 2s ago 12m 35.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (3m) 2s ago 8m 18.0M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (3m) 70s ago 8m 14.9M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (28s) 2s ago 12m 51.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (8s) 2s ago 11m 12.6M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (11m) 2s ago 11m 54.2M 2736M 17.2.0 e1d6a67b021e 68616c87205c 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (10m) 2s ago 10m 55.1M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (10m) 70s ago 10m 56.7M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:46:35.141 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (10m) 70s ago 10m 53.8M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (9m) 70s ago 9m 52.8M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (9m) 70s ago 9m 56.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (2m) 70s ago 8m 64.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (7m) 2s ago 7m 76.5M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (7m) 70s ago 7m 75.0M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (7m) 2s ago 7m 75.6M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:46:35.142 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (7m) 70s ago 7m 74.0M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-06-01T02:46:35.559 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 7 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:46:35.560 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[64430]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[64430]: pgmap v69: 161 pgs: 6 active+undersized+degraded, 21 active+undersized, 134 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41/723 objects degraded (5.671%) 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[64430]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[64430]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/2562455609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[66884]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:35.918 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[66884]: pgmap v69: 161 pgs: 6 active+undersized+degraded, 21 active+undersized, 134 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41/723 objects degraded (5.671%) 2024-06-01T02:46:35.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[66884]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:35.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[66884]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:35.919 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:35 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/2562455609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:46:35.919 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:46:35.920 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "7/23 daemons upgraded", 2024-06-01T02:46:35.920 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:46:35.920 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:46:35.920 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:46:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:35 smithi146 ceph-mon[57708]: from='client.34257 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:35 smithi146 ceph-mon[57708]: pgmap v69: 161 pgs: 6 active+undersized+degraded, 21 active+undersized, 134 active+clean; 585 KiB data, 136 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41/723 objects degraded (5.671%) 2024-06-01T02:46:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:35 smithi146 ceph-mon[57708]: from='client.44191 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:35 smithi146 ceph-mon[57708]: from='client.34269 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:35 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/2562455609' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN Degraded data redundancy: 41/723 objects degraded (5.671%), 6 pgs degraded 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 41/723 objects degraded (5.671%), 6 pgs degraded 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.0 is active+undersized+degraded, acting [2,6] 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.4 is active+undersized+degraded, acting [2,5] 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-06-01T02:46:36.339 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-06-01T02:46:37.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:36 smithi062 ceph-mon[64430]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:37.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:36 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/208827925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:46:37.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:36 smithi062 ceph-mon[66884]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:37.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:36 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/208827925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:46:37.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:36 smithi146 ceph-mon[57708]: from='client.34281 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:46:37.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:36 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/208827925' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 6 pgs degraded) 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:38.001 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[64430]: Upgrade: osd.2 is safe to restart 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 6 pgs degraded) 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:46:38.002 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-01T02:46:38.003 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:37 smithi062 ceph-mon[66884]: Upgrade: osd.2 is safe to restart 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: pgmap v70: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 6 pgs degraded) 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:38.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-06-01T02:46:38.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:37 smithi146 ceph-mon[57708]: Upgrade: osd.2 is safe to restart 2024-06-01T02:46:38.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-01T02:46:38.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:40.129 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[64430]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:46:40.129 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[64430]: Upgrade: Updating osd.2 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[64430]: Deploying daemon osd.2 on smithi062 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[66884]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[66884]: Upgrade: Updating osd.2 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:40.130 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:39 smithi062 ceph-mon[66884]: Deploying daemon osd.2 on smithi062 2024-06-01T02:46:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:39 smithi146 ceph-mon[57708]: pgmap v71: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:46:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:39 smithi146 ceph-mon[57708]: Upgrade: Updating osd.2 2024-06-01T02:46:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-01T02:46:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:39 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:40.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:39 smithi146 ceph-mon[57708]: Deploying daemon osd.2 on smithi062 2024-06-01T02:46:42.113 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:41 smithi062 ceph-mon[64430]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:46:42.114 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:41 smithi062 ceph-mon[66884]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:46:42.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:41 smithi146 ceph-mon[57708]: pgmap v72: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:46:42.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:42 smithi062 systemd[1]: Stopping Ceph osd.2 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:46:43.013 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[38513]: 2024-06-01T02:46:42.602+0000 7f9c0c316700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:46:43.013 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[38513]: 2024-06-01T02:46:42.602+0000 7f9c0c316700 -1 osd.2 116 *** Got signal Terminated *** 2024-06-01T02:46:43.013 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:42 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[38513]: 2024-06-01T02:46:42.602+0000 7f9c0c316700 -1 osd.2 116 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:46:43.939 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[64430]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:43.939 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[64430]: osd.2 marked itself down and dead 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[66884]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:43.940 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:43 smithi062 ceph-mon[66884]: osd.2 marked itself down and dead 2024-06-01T02:46:43.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:43 smithi146 ceph-mon[57708]: pgmap v73: 161 pgs: 161 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:43 smithi146 ceph-mon[57708]: osd.2 marked itself down and dead 2024-06-01T02:46:44.195 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:43 smithi062 podman[74529]: 2024-06-01 02:46:43.940022053 +0000 UTC m=+1.465994780 container died 68616c87205cc534d378c46f8f0e0e3cbfa61cb6b19a053cec435ec5c1dbee03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , name=centos-stream, vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, ceph=True, io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True) 2024-06-01T02:46:44.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:44 smithi062 podman[74529]: 2024-06-01 02:46:44.360570033 +0000 UTC m=+1.886542779 container cleanup 68616c87205cc534d378c46f8f0e0e3cbfa61cb6b19a053cec435ec5c1dbee03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-type=git, RELEASE=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, version=8, io.openshift.expose-services=) 2024-06-01T02:46:44.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:44 smithi062 bash[74529]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2 2024-06-01T02:46:44.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:44 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:44.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:44 smithi146 ceph-mon[57708]: osdmap e117: 8 total, 7 up, 8 in 2024-06-01T02:46:44.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:44 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:45.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:45.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-mon[64430]: osdmap e117: 8 total, 7 up, 8 in 2024-06-01T02:46:45.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:45.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:46:45.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-mon[66884]: osdmap e117: 8 total, 7 up, 8 in 2024-06-01T02:46:45.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:45.014 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:44 smithi062 podman[74544]: 2024-06-01 02:46:44.629098293 +0000 UTC m=+0.682851163 container remove 68616c87205cc534d378c46f8f0e0e3cbfa61cb6b19a053cec435ec5c1dbee03 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, io.k8s.display-name=CentOS Stream 8, vcs-type=git, distribution-scope=public, RELEASE=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, release=754, name=centos-stream) 2024-06-01T02:46:45.015 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:46:44] "GET /metrics HTTP/1.1" 200 38481 "" "Prometheus/2.33.4" 2024-06-01T02:46:45.365 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:45 smithi062 podman[74635]: 2024-06-01 02:46:45.054841694 +0000 UTC m=+0.112713301 container create a1592c916f2a0d38ce341b954c92a4ea2e4b473f53c533218f874f8f36086c59 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-06-01T02:46:45.365 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:45 smithi062 podman[74635]: 2024-06-01 02:46:44.959811243 +0000 UTC m=+0.017682876 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:45.365 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:45 smithi062 podman[74635]: 2024-06-01 02:46:45.265832912 +0000 UTC m=+0.323704519 container init a1592c916f2a0d38ce341b954c92a4ea2e4b473f53c533218f874f8f36086c59 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5) 2024-06-01T02:46:45.365 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:45 smithi062 podman[74635]: 2024-06-01 02:46:45.269489395 +0000 UTC m=+0.327361017 container start a1592c916f2a0d38ce341b954c92a4ea2e4b473f53c533218f874f8f36086c59 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-06-01T02:46:45.366 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:45 smithi062 podman[74635]: 2024-06-01 02:46:45.365954771 +0000 UTC m=+0.423826388 container attach a1592c916f2a0d38ce341b954c92a4ea2e4b473f53c533218f874f8f36086c59 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240528, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-01T02:46:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:45 smithi062 ceph-mon[64430]: pgmap v75: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:45.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:45 smithi062 ceph-mon[64430]: osdmap e118: 8 total, 7 up, 8 in 2024-06-01T02:46:45.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:45 smithi062 ceph-mon[66884]: pgmap v75: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:45.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:45 smithi062 ceph-mon[66884]: osdmap e118: 8 total, 7 up, 8 in 2024-06-01T02:46:45.764 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:45 smithi062 podman[74635]: 2024-06-01 02:46:45.582041516 +0000 UTC m=+0.639913130 container died a1592c916f2a0d38ce341b954c92a4ea2e4b473f53c533218f874f8f36086c59 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-01T02:46:45.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:45 smithi146 ceph-mon[57708]: pgmap v75: 161 pgs: 12 stale+active+clean, 149 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:46:45.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:45 smithi146 ceph-mon[57708]: osdmap e118: 8 total, 7 up, 8 in 2024-06-01T02:46:46.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 podman[74660]: 2024-06-01 02:46:46.103742757 +0000 UTC m=+0.515482807 container remove a1592c916f2a0d38ce341b954c92a4ea2e4b473f53c533218f874f8f36086c59 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-06-01T02:46:46.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.2.service: Deactivated successfully. 2024-06-01T02:46:46.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 systemd[1]: Stopped Ceph osd.2 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:46:46.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.2.service: Consumed 6.897s CPU time. 2024-06-01T02:46:46.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 systemd[1]: Starting Ceph osd.2 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:46:46.790 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:46 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 28/723 objects degraded (3.873%), 3 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:46.791 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:46 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 28/723 objects degraded (3.873%), 3 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:46.791 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 podman[74760]: 2024-06-01 02:46:46.538451573 +0000 UTC m=+0.099192969 container create d036f74db44e100870798ac796bf227b7923588bc5746a4910eaf513c002bdbc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-01T02:46:46.791 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 podman[74760]: 2024-06-01 02:46:46.456925489 +0000 UTC m=+0.017667129 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:46.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:46 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 28/723 objects degraded (3.873%), 3 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:47.253 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 podman[74760]: 2024-06-01 02:46:46.791187282 +0000 UTC m=+0.351928687 container init d036f74db44e100870798ac796bf227b7923588bc5746a4910eaf513c002bdbc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc) 2024-06-01T02:46:47.254 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 podman[74760]: 2024-06-01 02:46:46.794902642 +0000 UTC m=+0.355644048 container start d036f74db44e100870798ac796bf227b7923588bc5746a4910eaf513c002bdbc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-01T02:46:47.254 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:46 smithi062 podman[74760]: 2024-06-01 02:46:46.882561132 +0000 UTC m=+0.443302550 container attach d036f74db44e100870798ac796bf227b7923588bc5746a4910eaf513c002bdbc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:46:47.513 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 bash[74760]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-06-01T02:46:47.514 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-01T02:46:47.515 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate[74775]: --> ceph-volume raw activate successful for osd ID: 2 2024-06-01T02:46:47.515 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 podman[74760]: 2024-06-01 02:46:47.279144909 +0000 UTC m=+0.839886314 container died d036f74db44e100870798ac796bf227b7923588bc5746a4910eaf513c002bdbc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-01T02:46:47.878 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-mon[64430]: pgmap v77: 161 pgs: 3 active+undersized+degraded, 8 stale+active+clean, 11 active+undersized, 139 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 28/723 objects degraded (3.873%) 2024-06-01T02:46:47.878 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:47 smithi062 ceph-mon[66884]: pgmap v77: 161 pgs: 3 active+undersized+degraded, 8 stale+active+clean, 11 active+undersized, 139 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 28/723 objects degraded (3.873%) 2024-06-01T02:46:47.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:47 smithi146 ceph-mon[57708]: pgmap v77: 161 pgs: 3 active+undersized+degraded, 8 stale+active+clean, 11 active+undersized, 139 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 28/723 objects degraded (3.873%) 2024-06-01T02:46:48.155 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:47 smithi062 podman[74896]: 2024-06-01 02:46:47.878686718 +0000 UTC m=+0.594034511 container remove d036f74db44e100870798ac796bf227b7923588bc5746a4910eaf513c002bdbc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-06-01T02:46:48.155 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 podman[74956]: 2024-06-01 02:46:48.096205531 +0000 UTC m=+0.112395385 container create c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:46:48.155 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 podman[74956]: 2024-06-01 02:46:48.001495964 +0000 UTC m=+0.017685826 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:46:48.423 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 podman[74956]: 2024-06-01 02:46:48.323596172 +0000 UTC m=+0.339786035 container init c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-06-01T02:46:48.423 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 podman[74956]: 2024-06-01 02:46:48.326964066 +0000 UTC m=+0.343153923 container start c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:46:48.423 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 ceph-osd[74976]: -- 172.21.15.62:0/3759259368 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56207575f680 con 0x562075729000 2024-06-01T02:46:48.763 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 bash[74956]: c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 2024-06-01T02:46:48.763 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 systemd[1]: Started Ceph osd.2 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:46:48.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:49.263 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:46:48.870+0000 7f2349d0c7c0 -1 Falling back to public interface 2024-06-01T02:46:49.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:49.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:49.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:49 smithi062 ceph-mon[64430]: pgmap v78: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:49.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:49.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:49.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:49 smithi062 ceph-mon[66884]: pgmap v78: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:49 smithi146 ceph-mon[57708]: pgmap v78: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:51.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:51 smithi062 ceph-mon[64430]: pgmap v79: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:51.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:51 smithi062 ceph-mon[66884]: pgmap v79: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:51.763 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:46:51.588+0000 7f2349d0c7c0 -1 osd.2 0 read_superblock omap replica is missing. 2024-06-01T02:46:51.764 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:51 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:46:51.613+0000 7f2349d0c7c0 -1 osd.2 116 log_to_monitors true 2024-06-01T02:46:51.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:51 smithi146 ceph-mon[57708]: pgmap v79: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:52.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:52 smithi146 ceph-mon[57708]: from='osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:46:52.857 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:52 smithi146 ceph-mon[57708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:46:52.858 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:52 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 64/723 objects degraded (8.852%), 12 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:53.013 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:46:52.615+0000 7f233b163700 -1 osd.2 116 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:46:53.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-mon[64430]: from='osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:46:53.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-mon[64430]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:46:53.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 64/723 objects degraded (8.852%), 12 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:53.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-mon[66884]: from='osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:46:53.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-mon[66884]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-01T02:46:53.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:52 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 64/723 objects degraded (8.852%), 12 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:53.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: pgmap v80: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 64/723 objects degraded (8.852%) 2024-06-01T02:46:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T02:46:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: osdmap e119: 8 total, 7 up, 8 in 2024-06-01T02:46:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: from='osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:53.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: pgmap v80: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 64/723 objects degraded (8.852%) 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: osdmap e119: 8 total, 7 up, 8 in 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: from='osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: pgmap v80: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 140 MiB used, 715 GiB / 715 GiB avail; 64/723 objects degraded (8.852%) 2024-06-01T02:46:54.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-01T02:46:54.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: osdmap e119: 8 total, 7 up, 8 in 2024-06-01T02:46:54.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: from='osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:54.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:46:54.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.887 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:46:54] "GET /metrics HTTP/1.1" 200 38481 "" "Prometheus/2.33.4" 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727] boot 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: osdmap e120: 8 total, 8 up, 8 in 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[64430]: osdmap e121: 8 total, 8 up, 8 in 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:54.888 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727] boot 2024-06-01T02:46:54.889 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: osdmap e120: 8 total, 8 up, 8 in 2024-06-01T02:46:54.889 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:46:54.889 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:54.889 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.889 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.889 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:54 smithi062 ceph-mon[66884]: osdmap e121: 8 total, 8 up, 8 in 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: osd.2 [v2:172.21.15.62:6818/1943629727,v1:172.21.15.62:6819/1943629727] boot 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: osdmap e120: 8 total, 8 up, 8 in 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:54.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:54 smithi146 ceph-mon[57708]: osdmap e121: 8 total, 8 up, 8 in 2024-06-01T02:46:56.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:55 smithi146 ceph-mon[57708]: pgmap v83: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:56.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:55 smithi062 ceph-mon[64430]: pgmap v83: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:56.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:55 smithi062 ceph-mon[66884]: pgmap v83: 161 pgs: 12 active+undersized+degraded, 35 active+undersized, 114 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s; 64/723 objects degraded (8.852%) 2024-06-01T02:46:57.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:46:57.538+0000 7f15439f1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: pgmap v85: 161 pgs: 9 active+undersized+degraded, 28 active+undersized, 124 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/723 objects degraded (4.979%) 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 36/723 objects degraded (4.979%), 9 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:58.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: pgmap v85: 161 pgs: 9 active+undersized+degraded, 28 active+undersized, 124 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/723 objects degraded (4.979%) 2024-06-01T02:46:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 36/723 objects degraded (4.979%), 9 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: pgmap v85: 161 pgs: 9 active+undersized+degraded, 28 active+undersized, 124 active+clean; 585 KiB data, 156 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 36/723 objects degraded (4.979%) 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 36/723 objects degraded (4.979%), 9 pgs degraded (PG_DEGRADED) 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:46:58.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:46:58.266 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.266 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:46:58.266 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:46:58.921 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:46:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:46:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:46:59.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:58 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:46:59.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:58 smithi146 ceph-mon[57708]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-06-01T02:46:59.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:58 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:46:59.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:58 smithi062 ceph-mon[64430]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-06-01T02:46:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:58 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:46:59.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:58 smithi062 ceph-mon[66884]: Upgrade: unsafe to stop osd(s) at this time (15 PGs are or would become offline) 2024-06-01T02:47:00.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:59 smithi146 ceph-mon[57708]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-01T02:47:00.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:59 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/723 objects degraded (4.979%), 9 pgs degraded) 2024-06-01T02:47:00.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:46:59 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:47:00.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:59 smithi062 ceph-mon[64430]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-01T02:47:00.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:59 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/723 objects degraded (4.979%), 9 pgs degraded) 2024-06-01T02:47:00.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:46:59 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:47:00.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:59 smithi062 ceph-mon[66884]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-01T02:47:00.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:59 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 36/723 objects degraded (4.979%), 9 pgs degraded) 2024-06-01T02:47:00.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:46:59 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:47:02.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:01 smithi146 ceph-mon[57708]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:47:02.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:01 smithi062 ceph-mon[64430]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:47:02.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:01 smithi062 ceph-mon[66884]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:47:03.927 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:04.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:03 smithi146 ceph-mon[57708]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-06-01T02:47:04.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:03 smithi062 ceph-mon[64430]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-06-01T02:47:04.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:03 smithi062 ceph-mon[66884]: pgmap v88: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2024-06-01T02:47:04.934 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:47:04] "GET /metrics HTTP/1.1" 200 38348 "" "Prometheus/2.33.4" 2024-06-01T02:47:05.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:04 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:04 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:04 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:06.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:05 smithi146 ceph-mon[57708]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:47:06.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:05 smithi062 ceph-mon[64430]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:47:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:05 smithi062 ceph-mon[66884]: pgmap v89: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:47:06.718 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (3m) 14s ago 8m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (3m) 103s ago 8m 45.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (3m) 14s ago 7m 53.9M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (2m) 103s ago 13m 450M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (3m) 14s ago 14m 505M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (2m) 14s ago 14m 48.7M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (113s) 103s ago 13m 17.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (2m) 14s ago 13m 36.5M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (3m) 14s ago 9m 18.4M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (3m) 103s ago 9m 14.9M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:47:07.403 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (60s) 14s ago 12m 52.5M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (40s) 14s ago 12m 43.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (19s) 14s ago 11m 12.6M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (11m) 14s ago 11m 55.6M 2736M 17.2.0 e1d6a67b021e fbc5364b2028 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (11m) 103s ago 11m 56.7M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (10m) 103s ago 10m 53.8M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (10m) 103s ago 10m 52.8M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (9m) 103s ago 9m 56.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (2m) 103s ago 9m 64.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (7m) 14s ago 7m 76.6M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (7m) 103s ago 7m 75.0M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (7m) 14s ago 7m 75.7M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:47:07.404 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (7m) 103s ago 7m 74.0M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:07.820 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 8 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:47:07.821 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:47:08.171 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:47:08.171 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:47:08.171 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:47:08.171 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:47:08.171 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "8/23 daemons upgraded", 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:47:08.172 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:47:08.172 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[66884]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-06-01T02:47:08.172 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[66884]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:08.172 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[66884]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1496843276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[64430]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[64430]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[64430]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:08.173 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:07 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1496843276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:07 smithi146 ceph-mon[57708]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-06-01T02:47:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:07 smithi146 ceph-mon[57708]: from='client.44227 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:07 smithi146 ceph-mon[57708]: from='client.34305 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:07 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1496843276' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:08.583 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:47:08.946 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:08.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:09.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:08 smithi146 ceph-mon[57708]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:09.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:08 smithi146 ceph-mon[57708]: from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:09.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:08 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1767748397' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:47:09.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:08 smithi062 ceph-mon[64430]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:09.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:08 smithi062 ceph-mon[64430]: from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:09.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:08 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1767748397' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:47:09.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:08 smithi062 ceph-mon[66884]: from='client.34311 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:09.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:08 smithi062 ceph-mon[66884]: from='client.34323 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:09.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:08 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1767748397' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:47:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:09 smithi146 ceph-mon[57708]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:09 smithi062 ceph-mon[64430]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:10.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:09 smithi062 ceph-mon[66884]: pgmap v91: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:11 smithi146 ceph-mon[57708]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:11 smithi062 ceph-mon[64430]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:12.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:11 smithi062 ceph-mon[66884]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:47:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:13.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:47:13.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:13.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:47:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:13.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:13.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:14.247 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:13 smithi062 ceph-mon[64430]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:14.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:13 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:47:14.248 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:13 smithi062 ceph-mon[64430]: Upgrade: osd.3 is safe to restart 2024-06-01T02:47:14.248 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:13 smithi062 ceph-mon[66884]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:14.248 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:13 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:47:14.248 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:13 smithi062 ceph-mon[66884]: Upgrade: osd.3 is safe to restart 2024-06-01T02:47:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:13 smithi146 ceph-mon[57708]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:13 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-06-01T02:47:14.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:13 smithi146 ceph-mon[57708]: Upgrade: osd.3 is safe to restart 2024-06-01T02:47:14.921 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:47:14] "GET /metrics HTTP/1.1" 200 38493 "" "Prometheus/2.33.4" 2024-06-01T02:47:15.234 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:47:15.235 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:14 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:15.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:14 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:15.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:14 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:15.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:14 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-01T02:47:15.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:14 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:16.139 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:15 smithi062 ceph-mon[64430]: Upgrade: Updating osd.3 2024-06-01T02:47:16.139 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:15 smithi062 ceph-mon[64430]: Deploying daemon osd.3 on smithi062 2024-06-01T02:47:16.140 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:15 smithi062 ceph-mon[64430]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:16.140 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:15 smithi062 ceph-mon[66884]: Upgrade: Updating osd.3 2024-06-01T02:47:16.140 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:15 smithi062 ceph-mon[66884]: Deploying daemon osd.3 on smithi062 2024-06-01T02:47:16.140 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:15 smithi062 ceph-mon[66884]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:15 smithi146 ceph-mon[57708]: Upgrade: Updating osd.3 2024-06-01T02:47:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:15 smithi146 ceph-mon[57708]: Deploying daemon osd.3 on smithi062 2024-06-01T02:47:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:15 smithi146 ceph-mon[57708]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:17.868 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:17 smithi062 systemd[1]: Stopping Ceph osd.3 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:47:18.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-mon[66884]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:18.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-mon[66884]: osd.3 marked itself down and dead 2024-06-01T02:47:18.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-mon[64430]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:18.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-mon[64430]: osd.3 marked itself down and dead 2024-06-01T02:47:18.264 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[41479]: 2024-06-01T02:47:17.867+0000 7f392021e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:47:18.264 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[41479]: 2024-06-01T02:47:17.867+0000 7f392021e700 -1 osd.3 121 *** Got signal Terminated *** 2024-06-01T02:47:18.264 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:17 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[41479]: 2024-06-01T02:47:17.867+0000 7f392021e700 -1 osd.3 121 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:47:18.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:17 smithi146 ceph-mon[57708]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:18.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:17 smithi146 ceph-mon[57708]: osd.3 marked itself down and dead 2024-06-01T02:47:18.636 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:18 smithi062 podman[76956]: 2024-06-01 02:47:18.317604316 +0000 UTC m=+0.543794670 container died fbc5364b20289b4d2dc782f5c818d8ac1af5f6757c1f61f96d306c28c571f1f4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, RELEASE=HEAD, architecture=x86_64, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-type=git, release=754, name=centos-stream, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., version=8) 2024-06-01T02:47:18.936 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:18 smithi062 podman[76956]: 2024-06-01 02:47:18.75173397 +0000 UTC m=+0.977924330 container cleanup fbc5364b20289b4d2dc782f5c818d8ac1af5f6757c1f61f96d306c28c571f1f4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, release=754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-06-01T02:47:18.936 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:18 smithi062 bash[76956]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3 2024-06-01T02:47:18.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:18 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:47:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:18 smithi062 ceph-mon[64430]: osdmap e122: 8 total, 7 up, 8 in 2024-06-01T02:47:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:18 smithi062 ceph-mon[64430]: osdmap e123: 8 total, 7 up, 8 in 2024-06-01T02:47:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:18 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:47:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:18 smithi062 ceph-mon[66884]: osdmap e122: 8 total, 7 up, 8 in 2024-06-01T02:47:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:18 smithi062 ceph-mon[66884]: osdmap e123: 8 total, 7 up, 8 in 2024-06-01T02:47:19.264 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:18 smithi062 podman[76974]: 2024-06-01 02:47:18.936441187 +0000 UTC m=+0.614672819 container remove fbc5364b20289b4d2dc782f5c818d8ac1af5f6757c1f61f96d306c28c571f1f4 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, RELEASE=HEAD, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-type=git, io.openshift.expose-services=, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-06-01T02:47:19.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:18 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:47:19.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:18 smithi146 ceph-mon[57708]: osdmap e122: 8 total, 7 up, 8 in 2024-06-01T02:47:19.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:18 smithi146 ceph-mon[57708]: osdmap e123: 8 total, 7 up, 8 in 2024-06-01T02:47:19.706 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:19 smithi062 podman[77064]: 2024-06-01 02:47:19.395678827 +0000 UTC m=+0.113457658 container create 5eee40a636002770cba49677d16312954dfe9f223738ba0838f24fe2b38995e8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=) 2024-06-01T02:47:19.706 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:19 smithi062 podman[77064]: 2024-06-01 02:47:19.301067782 +0000 UTC m=+0.018846630 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:47:19.706 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:19 smithi062 podman[77064]: 2024-06-01 02:47:19.615723828 +0000 UTC m=+0.333502666 container init 5eee40a636002770cba49677d16312954dfe9f223738ba0838f24fe2b38995e8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:47:19.706 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:19 smithi062 podman[77064]: 2024-06-01 02:47:19.61954007 +0000 UTC m=+0.337318898 container start 5eee40a636002770cba49677d16312954dfe9f223738ba0838f24fe2b38995e8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-01T02:47:19.707 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:19 smithi062 podman[77064]: 2024-06-01 02:47:19.706648168 +0000 UTC m=+0.424427013 container attach 5eee40a636002770cba49677d16312954dfe9f223738ba0838f24fe2b38995e8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2) 2024-06-01T02:47:19.986 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:19 smithi062 podman[77064]: 2024-06-01 02:47:19.926824542 +0000 UTC m=+0.644603369 container died 5eee40a636002770cba49677d16312954dfe9f223738ba0838f24fe2b38995e8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-01T02:47:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:19 smithi062 ceph-mon[66884]: pgmap v97: 161 pgs: 31 peering, 14 stale+active+clean, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:19 smithi062 ceph-mon[66884]: Health check failed: Reduced data availability: 5 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-06-01T02:47:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:19 smithi062 ceph-mon[64430]: pgmap v97: 161 pgs: 31 peering, 14 stale+active+clean, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:19 smithi062 ceph-mon[64430]: Health check failed: Reduced data availability: 5 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-06-01T02:47:20.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:19 smithi146 ceph-mon[57708]: pgmap v97: 161 pgs: 31 peering, 14 stale+active+clean, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:20.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:19 smithi146 ceph-mon[57708]: Health check failed: Reduced data availability: 5 pgs inactive, 10 pgs peering (PG_AVAILABILITY) 2024-06-01T02:47:20.763 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 podman[77090]: 2024-06-01 02:47:20.535939144 +0000 UTC m=+0.599073911 container remove 5eee40a636002770cba49677d16312954dfe9f223738ba0838f24fe2b38995e8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-01T02:47:20.763 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.3.service: Deactivated successfully. 2024-06-01T02:47:20.763 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 systemd[1]: Stopped Ceph osd.3 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:47:20.763 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.3.service: Consumed 8.921s CPU time. 2024-06-01T02:47:21.119 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 systemd[1]: Starting Ceph osd.3 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:47:21.120 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 podman[77189]: 2024-06-01 02:47:20.895255818 +0000 UTC m=+0.017897193 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:47:21.120 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:20 smithi062 podman[77189]: 2024-06-01 02:47:20.995287446 +0000 UTC m=+0.117928812 container create 42c36191c0a9903da39ceba86015d392305cb3210e97ae25fc97b96a1f5a493e (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS) 2024-06-01T02:47:21.513 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 podman[77189]: 2024-06-01 02:47:21.248179488 +0000 UTC m=+0.370820852 container init 42c36191c0a9903da39ceba86015d392305cb3210e97ae25fc97b96a1f5a493e (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:47:21.513 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 podman[77189]: 2024-06-01 02:47:21.251751435 +0000 UTC m=+0.374392799 container start 42c36191c0a9903da39ceba86015d392305cb3210e97ae25fc97b96a1f5a493e (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True) 2024-06-01T02:47:21.513 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 podman[77189]: 2024-06-01 02:47:21.33984155 +0000 UTC m=+0.462482942 container attach 42c36191c0a9903da39ceba86015d392305cb3210e97ae25fc97b96a1f5a493e (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-01T02:47:21.989 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-01T02:47:21.989 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-01T02:47:21.989 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-01T02:47:21.989 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-01T02:47:21.989 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-01T02:47:21.989 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 bash[77189]: --> ceph-volume raw activate successful for osd ID: 3 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate[77204]: --> ceph-volume raw activate successful for osd ID: 3 2024-06-01T02:47:21.990 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:21 smithi062 podman[77189]: 2024-06-01 02:47:21.742611698 +0000 UTC m=+0.865253066 container died 42c36191c0a9903da39ceba86015d392305cb3210e97ae25fc97b96a1f5a493e (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528) 2024-06-01T02:47:22.258 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:21 smithi146 ceph-mon[57708]: pgmap v99: 161 pgs: 31 peering, 14 stale+active+clean, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:22.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-mon[64430]: pgmap v99: 161 pgs: 31 peering, 14 stale+active+clean, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:21 smithi062 ceph-mon[66884]: pgmap v99: 161 pgs: 31 peering, 14 stale+active+clean, 116 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:22.683 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 podman[77340]: 2024-06-01 02:47:22.327524628 +0000 UTC m=+0.579050671 container remove 42c36191c0a9903da39ceba86015d392305cb3210e97ae25fc97b96a1f5a493e (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-01T02:47:22.683 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 podman[77399]: 2024-06-01 02:47:22.44564707 +0000 UTC m=+0.017620573 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:47:22.683 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 podman[77399]: 2024-06-01 02:47:22.553511821 +0000 UTC m=+0.125485362 container create 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-01T02:47:23.013 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 podman[77399]: 2024-06-01 02:47:22.790077841 +0000 UTC m=+0.362051340 container init 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, GIT_CLEAN=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-01T02:47:23.013 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 podman[77399]: 2024-06-01 02:47:22.793679887 +0000 UTC m=+0.365653389 container start 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-06-01T02:47:23.013 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 ceph-osd[77418]: -- 172.21.15.62:0/2304692841 <== mon.0 v2:172.21.15.62:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56510bcdf680 con 0x56510bca9400 2024-06-01T02:47:23.013 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 bash[77399]: 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d 2024-06-01T02:47:23.013 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:22 smithi062 systemd[1]: Started Ceph osd.3 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:47:23.647 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:47:23.334+0000 7f5ba9d587c0 -1 Falling back to public interface 2024-06-01T02:47:23.897 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:24.009 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[64430]: pgmap v100: 161 pgs: 11 active+undersized+degraded, 31 peering, 3 stale+active+clean, 16 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 52/723 objects degraded (7.192%) 2024-06-01T02:47:24.009 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:24.009 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:24.009 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 52/723 objects degraded (7.192%), 11 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:24.009 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[66884]: pgmap v100: 161 pgs: 11 active+undersized+degraded, 31 peering, 3 stale+active+clean, 16 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 52/723 objects degraded (7.192%) 2024-06-01T02:47:24.010 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:24.010 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:24.010 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:23 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 52/723 objects degraded (7.192%), 11 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:23 smithi146 ceph-mon[57708]: pgmap v100: 161 pgs: 11 active+undersized+degraded, 31 peering, 3 stale+active+clean, 16 active+undersized, 100 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 52/723 objects degraded (7.192%) 2024-06-01T02:47:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:23 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:23 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:23 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 52/723 objects degraded (7.192%), 11 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:24.903 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:47:24] "GET /metrics HTTP/1.1" 200 38493 "" "Prometheus/2.33.4" 2024-06-01T02:47:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:24 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:25.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:24 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:25.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:24 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:26.160 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:25 smithi062 ceph-mon[64430]: pgmap v101: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-06-01T02:47:26.161 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:25 smithi062 ceph-mon[64430]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-06-01T02:47:26.161 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:25 smithi062 ceph-mon[66884]: pgmap v101: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-06-01T02:47:26.161 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:25 smithi062 ceph-mon[66884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-06-01T02:47:26.161 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:47:26.041+0000 7f5ba9d587c0 -1 osd.3 0 read_superblock omap replica is missing. 2024-06-01T02:47:26.161 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:26 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:47:26.073+0000 7f5ba9d587c0 -1 osd.3 121 log_to_monitors true 2024-06-01T02:47:26.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:25 smithi146 ceph-mon[57708]: pgmap v101: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-06-01T02:47:26.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:25 smithi146 ceph-mon[57708]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 10 pgs peering) 2024-06-01T02:47:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:26 smithi146 ceph-mon[57708]: from='osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:47:27.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:26 smithi146 ceph-mon[57708]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:47:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:26 smithi062 ceph-mon[64430]: from='osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:47:27.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:26 smithi062 ceph-mon[64430]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:47:27.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:26 smithi062 ceph-mon[66884]: from='osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:47:27.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:26 smithi062 ceph-mon[66884]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-01T02:47:27.669 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:47:27.398+0000 7f5b971a7700 -1 osd.3 121 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:47:28.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: pgmap v102: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-06-01T02:47:28.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T02:47:28.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: osdmap e124: 8 total, 7 up, 8 in 2024-06-01T02:47:28.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:47:28.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:47:28.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='osd.3 ' entity='osd.3' 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: pgmap v102: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: osdmap e124: 8 total, 7 up, 8 in 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='osd.3 ' entity='osd.3' 2024-06-01T02:47:28.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: pgmap v102: 161 pgs: 26 active+undersized+degraded, 40 active+undersized, 95 active+clean; 585 KiB data, 157 MiB used, 715 GiB / 715 GiB avail; 97/723 objects degraded (13.416%) 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: osdmap e124: 8 total, 7 up, 8 in 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi062", "root=default"]}]: dispatch 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='osd.3 ' entity='osd.3' 2024-06-01T02:47:28.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:28.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:28.924 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:28.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:29.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:47:29.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-mon[57708]: osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197] boot 2024-06-01T02:47:29.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-mon[57708]: osdmap e125: 8 total, 8 up, 8 in 2024-06-01T02:47:29.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:47:29.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:29.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:29.215 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:47:29.215 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[64430]: osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197] boot 2024-06-01T02:47:29.215 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[64430]: osdmap e125: 8 total, 8 up, 8 in 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[66884]: osd.3 [v2:172.21.15.62:6826/2899772197,v1:172.21.15.62:6827/2899772197] boot 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[66884]: osdmap e125: 8 total, 8 up, 8 in 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:29.216 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:30.181 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:29 smithi062 ceph-mon[64430]: pgmap v105: 161 pgs: 18 peering, 17 active+undersized+degraded, 31 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-06-01T02:47:30.181 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:29 smithi062 ceph-mon[64430]: osdmap e126: 8 total, 8 up, 8 in 2024-06-01T02:47:30.181 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:29 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 74/723 objects degraded (10.235%), 17 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:30.182 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:29 smithi062 ceph-mon[66884]: pgmap v105: 161 pgs: 18 peering, 17 active+undersized+degraded, 31 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-06-01T02:47:30.182 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:29 smithi062 ceph-mon[66884]: osdmap e126: 8 total, 8 up, 8 in 2024-06-01T02:47:30.182 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:29 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 74/723 objects degraded (10.235%), 17 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:29 smithi146 ceph-mon[57708]: pgmap v105: 161 pgs: 18 peering, 17 active+undersized+degraded, 31 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-06-01T02:47:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:29 smithi146 ceph-mon[57708]: osdmap e126: 8 total, 8 up, 8 in 2024-06-01T02:47:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:29 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 74/723 objects degraded (10.235%), 17 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:31 smithi146 ceph-mon[57708]: pgmap v107: 161 pgs: 18 peering, 17 active+undersized+degraded, 31 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-06-01T02:47:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:31 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:31 smithi062 ceph-mon[64430]: pgmap v107: 161 pgs: 18 peering, 17 active+undersized+degraded, 31 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-06-01T02:47:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:31 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:32.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:31 smithi062 ceph-mon[66884]: pgmap v107: 161 pgs: 18 peering, 17 active+undersized+degraded, 31 active+undersized, 95 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 74/723 objects degraded (10.235%) 2024-06-01T02:47:32.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:31 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:32.264 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:47:32.087+0000 7f15439f1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:47:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:47:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:47:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:33.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:33.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:32 smithi146 ceph-mon[57708]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[64430]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:33.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:33.265 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:32 smithi062 ceph-mon[66884]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:47:33.946 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:33.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:34.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:33 smithi146 ceph-mon[57708]: pgmap v108: 161 pgs: 18 peering, 1 active+undersized+degraded, 4 active+undersized, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 2/723 objects degraded (0.277%) 2024-06-01T02:47:34.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:33 smithi062 ceph-mon[64430]: pgmap v108: 161 pgs: 18 peering, 1 active+undersized+degraded, 4 active+undersized, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 2/723 objects degraded (0.277%) 2024-06-01T02:47:34.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:33 smithi062 ceph-mon[66884]: pgmap v108: 161 pgs: 18 peering, 1 active+undersized+degraded, 4 active+undersized, 138 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 2/723 objects degraded (0.277%) 2024-06-01T02:47:34.951 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:47:34] "GET /metrics HTTP/1.1" 200 38352 "" "Prometheus/2.33.4" 2024-06-01T02:47:35.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:34 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:34 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:35.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:34 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:35 smithi146 ceph-mon[57708]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:35 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/723 objects degraded (0.277%), 1 pg degraded) 2024-06-01T02:47:36.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:35 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:47:36.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:35 smithi062 ceph-mon[64430]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:36.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:35 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/723 objects degraded (0.277%), 1 pg degraded) 2024-06-01T02:47:36.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:35 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:47:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:35 smithi062 ceph-mon[66884]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:35 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/723 objects degraded (0.277%), 1 pg degraded) 2024-06-01T02:47:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:35 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:47:38.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:37 smithi062 ceph-mon[64430]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:38.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:37 smithi062 ceph-mon[66884]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:38.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:37 smithi146 ceph-mon[57708]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:47:38.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:38.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-01T02:47:38.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:38.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:38.962 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (4m) 12s ago 8m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (4m) 2m ago 8m 45.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (3m) 12s ago 7m 54.1M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (3m) 2m ago 13m 450M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (3m) 12s ago 15m 507M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (3m) 12s ago 15m 50.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (2m) 2m ago 13m 17.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (2m) 12s ago 13m 37.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (4m) 12s ago 9m 18.2M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (4m) 2m ago 9m 14.9M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:47:39.680 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (92s) 12s ago 13m 53.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (72s) 12s ago 12m 44.4M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (51s) 12s ago 12m 42.0M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (17s) 12s ago 12m 11.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (11m) 2m ago 11m 56.7M 2992M 17.2.0 e1d6a67b021e 97bc6ce5d8f7 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (11m) 2m ago 11m 53.8M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (10m) 2m ago 10m 52.8M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (10m) 2m ago 10m 56.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (3m) 2m ago 9m 64.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (8m) 12s ago 8m 76.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (8m) 2m ago 8m 75.0M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (8m) 12s ago 8m 75.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:47:39.681 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (8m) 2m ago 8m 74.0M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 4 2024-06-01T02:47:40.114 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 9 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:47:40.115 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:47:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:39 smithi062 ceph-mon[64430]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-06-01T02:47:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:39 smithi062 ceph-mon[64430]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:40.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:39 smithi062 ceph-mon[66884]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-06-01T02:47:40.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:39 smithi062 ceph-mon[66884]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:40.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:39 smithi146 ceph-mon[57708]: pgmap v111: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 307 B/s rd, 0 op/s 2024-06-01T02:47:40.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:39 smithi146 ceph-mon[57708]: from='client.44269 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "9/23 daemons upgraded", 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:47:40.465 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:47:40.877 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[66884]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[66884]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/832591683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/718602579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[64430]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[64430]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:41.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/832591683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:41.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:40 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/718602579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:47:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:40 smithi146 ceph-mon[57708]: from='client.34347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:40 smithi146 ceph-mon[57708]: from='client.44281 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:40 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/832591683' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:47:41.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:40 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/718602579' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:47:42.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:41 smithi062 ceph-mon[64430]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:42.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:41 smithi062 ceph-mon[64430]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 275 B/s rd, 0 op/s 2024-06-01T02:47:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:41 smithi062 ceph-mon[66884]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:42.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:41 smithi062 ceph-mon[66884]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 275 B/s rd, 0 op/s 2024-06-01T02:47:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:41 smithi146 ceph-mon[57708]: from='client.34365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:47:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:41 smithi146 ceph-mon[57708]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 275 B/s rd, 0 op/s 2024-06-01T02:47:43.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:43.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:43 smithi146 ceph-mon[57708]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-06-01T02:47:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:43.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[64430]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-06-01T02:47:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:44.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[66884]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-06-01T02:47:44.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:44.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:44.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:44.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:44 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:45.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:44 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:45.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:47:44] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.33.4" 2024-06-01T02:47:45.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:44 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:45.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:45 smithi146 ceph-mon[57708]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:46.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:45 smithi062 ceph-mon[64430]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:46.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:45 smithi062 ceph-mon[66884]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:47.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:47 smithi146 ceph-mon[57708]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:47.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:47 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:47.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:47 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:47.880 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:47 smithi146 ceph-mon[57708]: Upgrade: osd.4 is safe to restart 2024-06-01T02:47:48.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[64430]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:48.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:48.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:48.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[64430]: Upgrade: osd.4 is safe to restart 2024-06-01T02:47:48.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[66884]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:48.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:48.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-06-01T02:47:48.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:47 smithi062 ceph-mon[66884]: Upgrade: osd.4 is safe to restart 2024-06-01T02:47:48.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:48 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:48.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:49 smithi146 ceph-mon[57708]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:49 smithi146 ceph-mon[57708]: Upgrade: Updating osd.4 2024-06-01T02:47:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:47:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:49 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:49.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:49 smithi146 ceph-mon[57708]: Deploying daemon osd.4 on smithi146 2024-06-01T02:47:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[64430]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[64430]: Upgrade: Updating osd.4 2024-06-01T02:47:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:47:50.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[64430]: Deploying daemon osd.4 on smithi146 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[66884]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[66884]: Upgrade: Updating osd.4 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:47:50.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:49 smithi062 ceph-mon[66884]: Deploying daemon osd.4 on smithi146 2024-06-01T02:47:51.932 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:51 smithi146 ceph-mon[57708]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:52.194 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:51 smithi146 systemd[1]: Stopping Ceph osd.4 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:47:52.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:51 smithi062 ceph-mon[64430]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:52.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:51 smithi062 ceph-mon[66884]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:47:52.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:52 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[29294]: 2024-06-01T02:47:52.193+0000 7fdd7f81d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:47:52.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:52 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[29294]: 2024-06-01T02:47:52.193+0000 7fdd7f81d700 -1 osd.4 126 *** Got signal Terminated *** 2024-06-01T02:47:52.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:52 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[29294]: 2024-06-01T02:47:52.193+0000 7fdd7f81d700 -1 osd.4 126 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:47:53.154 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:52 smithi146 ceph-mon[57708]: osd.4 marked itself down and dead 2024-06-01T02:47:53.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:52 smithi062 ceph-mon[64430]: osd.4 marked itself down and dead 2024-06-01T02:47:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:52 smithi062 ceph-mon[66884]: osd.4 marked itself down and dead 2024-06-01T02:47:53.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:53 smithi146 podman[60401]: 2024-06-01 02:47:53.155189329 +0000 UTC m=+1.070854162 container died 97bc6ce5d8f75cf86e58a40341a02ac6974b40512a29b8302434e3f989f17a73 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-type=git, name=centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , architecture=x86_64, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.component=centos-stream-container) 2024-06-01T02:47:53.796 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:53 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:53.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:53.796 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:53 smithi146 podman[60401]: 2024-06-01 02:47:53.610810868 +0000 UTC m=+1.526475695 container cleanup 97bc6ce5d8f75cf86e58a40341a02ac6974b40512a29b8302434e3f989f17a73 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, name=centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vcs-type=git, architecture=x86_64, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8) 2024-06-01T02:47:53.796 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:53 smithi146 bash[60401]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4 2024-06-01T02:47:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:53 smithi062 ceph-mon[66884]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:53 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:47:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:53 smithi062 ceph-mon[66884]: osdmap e127: 8 total, 7 up, 8 in 2024-06-01T02:47:54.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:53 smithi062 ceph-mon[64430]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:54.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:53 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:47:54.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:53 smithi062 ceph-mon[64430]: osdmap e127: 8 total, 7 up, 8 in 2024-06-01T02:47:54.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:53 smithi146 ceph-mon[57708]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 173 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:47:54.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:53 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:47:54.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:53 smithi146 ceph-mon[57708]: osdmap e127: 8 total, 7 up, 8 in 2024-06-01T02:47:54.206 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:53 smithi146 podman[60417]: 2024-06-01 02:47:53.946459951 +0000 UTC m=+0.783352829 container remove 97bc6ce5d8f75cf86e58a40341a02ac6974b40512a29b8302434e3f989f17a73 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, release=754, RELEASE=HEAD, GIT_BRANCH=HEAD, distribution-scope=public, ceph=True, GIT_CLEAN=True, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-06-01T02:47:54.590 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:54 smithi146 podman[60504]: 2024-06-01 02:47:54.195586623 +0000 UTC m=+0.017335072 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:47:54.590 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:54 smithi146 podman[60504]: 2024-06-01 02:47:54.330180838 +0000 UTC m=+0.151929260 container create 86f8be6245d32eedad505d3329451190cbe701034bf21f8b09d590a1f563164f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-06-01T02:47:54.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:54 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:54.891 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:54 smithi146 ceph-mon[57708]: osdmap e128: 8 total, 7 up, 8 in 2024-06-01T02:47:54.892 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:54 smithi146 podman[60504]: 2024-06-01 02:47:54.590756618 +0000 UTC m=+0.412505063 container init 86f8be6245d32eedad505d3329451190cbe701034bf21f8b09d590a1f563164f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-06-01T02:47:54.892 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:54 smithi146 podman[60504]: 2024-06-01 02:47:54.594221178 +0000 UTC m=+0.415969614 container start 86f8be6245d32eedad505d3329451190cbe701034bf21f8b09d590a1f563164f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS) 2024-06-01T02:47:54.892 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:54 smithi146 podman[60504]: 2024-06-01 02:47:54.699187671 +0000 UTC m=+0.520936123 container attach 86f8be6245d32eedad505d3329451190cbe701034bf21f8b09d590a1f563164f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-01T02:47:55.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:54 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:55.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:54 smithi062 ceph-mon[64430]: osdmap e128: 8 total, 7 up, 8 in 2024-06-01T02:47:55.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:54 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:47:55.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:54 smithi062 ceph-mon[66884]: osdmap e128: 8 total, 7 up, 8 in 2024-06-01T02:47:55.014 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:47:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:47:54] "GET /metrics HTTP/1.1" 200 38499 "" "Prometheus/2.33.4" 2024-06-01T02:47:55.206 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:54 smithi146 podman[60504]: 2024-06-01 02:47:54.89189605 +0000 UTC m=+0.713644481 container died 86f8be6245d32eedad505d3329451190cbe701034bf21f8b09d590a1f563164f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-01T02:47:55.803 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:55 smithi146 ceph-mon[57708]: pgmap v121: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:47:55.803 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:55 smithi146 podman[60529]: 2024-06-01 02:47:55.552941048 +0000 UTC m=+0.655547649 container remove 86f8be6245d32eedad505d3329451190cbe701034bf21f8b09d590a1f563164f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:47:55.803 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:55 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.4.service: Deactivated successfully. 2024-06-01T02:47:55.803 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:55 smithi146 systemd[1]: Stopped Ceph osd.4 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:47:55.803 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:55 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.4.service: Consumed 9.091s CPU time. 2024-06-01T02:47:55.803 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:55 smithi146 systemd[1]: Starting Ceph osd.4 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:47:56.129 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:55 smithi146 ceph-mon[57708]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-06-01T02:47:56.129 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:55 smithi146 podman[60629]: 2024-06-01 02:47:55.901642311 +0000 UTC m=+0.017770250 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:47:56.129 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 podman[60629]: 2024-06-01 02:47:56.012187456 +0000 UTC m=+0.128315410 container create e8f223a341cb1afb0129dd294cd7f5765490e0d1c90342aa20285533af3b1e99 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-06-01T02:47:56.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:55 smithi062 ceph-mon[64430]: pgmap v121: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:47:56.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:55 smithi062 ceph-mon[64430]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-06-01T02:47:56.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:55 smithi062 ceph-mon[66884]: pgmap v121: 161 pgs: 32 peering, 14 stale+active+clean, 115 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:47:56.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:55 smithi062 ceph-mon[66884]: Health check failed: Reduced data availability: 6 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-06-01T02:47:56.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 podman[60629]: 2024-06-01 02:47:56.264860797 +0000 UTC m=+0.380988719 container init e8f223a341cb1afb0129dd294cd7f5765490e0d1c90342aa20285533af3b1e99 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:47:56.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 podman[60629]: 2024-06-01 02:47:56.268312837 +0000 UTC m=+0.384440769 container start e8f223a341cb1afb0129dd294cd7f5765490e0d1c90342aa20285533af3b1e99 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-06-01T02:47:56.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 podman[60629]: 2024-06-01 02:47:56.348173448 +0000 UTC m=+0.464301381 container attach e8f223a341cb1afb0129dd294cd7f5765490e0d1c90342aa20285533af3b1e99 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-01T02:47:57.092 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 47/723 objects degraded (6.501%), 9 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:57.092 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-01T02:47:57.092 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-01T02:47:57.092 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 bash[60629]: --> ceph-volume raw activate successful for osd ID: 4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate[60644]: --> ceph-volume raw activate successful for osd ID: 4 2024-06-01T02:47:57.093 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:56 smithi146 podman[60629]: 2024-06-01 02:47:56.747664074 +0000 UTC m=+0.863792007 container died e8f223a341cb1afb0129dd294cd7f5765490e0d1c90342aa20285533af3b1e99 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.33.5) 2024-06-01T02:47:57.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:56 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 47/723 objects degraded (6.501%), 9 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:57.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:56 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 47/723 objects degraded (6.501%), 9 pgs degraded (PG_DEGRADED) 2024-06-01T02:47:57.688 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 podman[60776]: 2024-06-01 02:47:57.376861577 +0000 UTC m=+0.623278541 container remove e8f223a341cb1afb0129dd294cd7f5765490e0d1c90342aa20285533af3b1e99 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD) 2024-06-01T02:47:57.688 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 podman[60836]: 2024-06-01 02:47:57.585957467 +0000 UTC m=+0.109793160 container create b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-01T02:47:57.688 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 podman[60836]: 2024-06-01 02:47:57.493718212 +0000 UTC m=+0.017553913 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:47:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:57 smithi146 ceph-mon[57708]: pgmap v122: 161 pgs: 9 active+undersized+degraded, 32 peering, 9 stale+active+clean, 11 active+undersized, 100 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 47/723 objects degraded (6.501%) 2024-06-01T02:47:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:58.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 podman[60836]: 2024-06-01 02:47:57.787996128 +0000 UTC m=+0.311831820 container init b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-01T02:47:58.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 podman[60836]: 2024-06-01 02:47:57.791203538 +0000 UTC m=+0.315039231 container start b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-01T02:47:58.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 ceph-osd[60854]: -- 172.21.15.146:0/2258066539 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55978bb03680 con 0x55978baccc00 2024-06-01T02:47:58.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 bash[60836]: b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 2024-06-01T02:47:58.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:57 smithi146 systemd[1]: Started Ceph osd.4 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:47:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:57 smithi062 ceph-mon[64430]: pgmap v122: 161 pgs: 9 active+undersized+degraded, 32 peering, 9 stale+active+clean, 11 active+undersized, 100 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 47/723 objects degraded (6.501%) 2024-06-01T02:47:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:58.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:57 smithi062 ceph-mon[66884]: pgmap v122: 161 pgs: 9 active+undersized+degraded, 32 peering, 9 stale+active+clean, 11 active+undersized, 100 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 47/723 objects degraded (6.501%) 2024-06-01T02:47:58.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:58.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:47:58.658 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:47:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:47:58.334+0000 7f86bad917c0 -1 Falling back to public interface 2024-06-01T02:47:58.944 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:47:58 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:47:58.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:47:59.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:59.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:59.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:59.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:47:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:00.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:59 smithi062 ceph-mon[64430]: pgmap v123: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:00.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:47:59 smithi062 ceph-mon[64430]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-06-01T02:48:00.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:59 smithi062 ceph-mon[66884]: pgmap v123: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:00.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:47:59 smithi062 ceph-mon[66884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-06-01T02:48:00.304 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:59 smithi146 ceph-mon[57708]: pgmap v123: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:00.304 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:47:59 smithi146 ceph-mon[57708]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 9 pgs peering) 2024-06-01T02:48:01.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:48:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:48:01.043+0000 7f86bad917c0 -1 osd.4 0 read_superblock omap replica is missing. 2024-06-01T02:48:01.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:48:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:48:01.078+0000 7f86bad917c0 -1 osd.4 126 log_to_monitors true 2024-06-01T02:48:01.985 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:01 smithi146 ceph-mon[57708]: pgmap v124: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:01.986 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:01 smithi146 ceph-mon[57708]: from='osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:48:02.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:01 smithi062 ceph-mon[64430]: pgmap v124: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:02.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:01 smithi062 ceph-mon[64430]: from='osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:48:02.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:01 smithi062 ceph-mon[64430]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:48:02.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:01 smithi062 ceph-mon[66884]: pgmap v124: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 174 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:02.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:01 smithi062 ceph-mon[66884]: from='osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:48:02.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:01 smithi062 ceph-mon[66884]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:48:02.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:01 smithi146 ceph-mon[57708]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-01T02:48:02.956 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:48:02.464+0000 7f86a81e0700 -1 osd.4 126 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:48:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T02:48:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: osdmap e129: 8 total, 7 up, 8 in 2024-06-01T02:48:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: from='osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:03.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 127/723 objects degraded (17.566%), 27 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[64430]: from='osd.4 ' entity='osd.4' 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: osdmap e129: 8 total, 7 up, 8 in 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: from='osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 127/723 objects degraded (17.566%), 27 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:02 smithi062 ceph-mon[66884]: from='osd.4 ' entity='osd.4' 2024-06-01T02:48:03.430 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: osdmap e129: 8 total, 7 up, 8 in 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: from='osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 127/723 objects degraded (17.566%), 27 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:03.431 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:02 smithi146 ceph-mon[57708]: from='osd.4 ' entity='osd.4' 2024-06-01T02:48:03.706 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:48:03.504Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:48:04.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: pgmap v126: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:04.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:48:04.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254] boot 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: osdmap e130: 8 total, 8 up, 8 in 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: pgmap v126: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254] boot 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: osdmap e130: 8 total, 8 up, 8 in 2024-06-01T02:48:04.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:03 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: pgmap v126: 161 pgs: 27 active+undersized+degraded, 41 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 127/723 objects degraded (17.566%) 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: osd.4 [v2:172.21.15.146:6800/3161316254,v1:172.21.15.146:6801/3161316254] boot 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: osdmap e130: 8 total, 8 up, 8 in 2024-06-01T02:48:04.265 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-01T02:48:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:04.266 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:03 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:04.996 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:48:04] "GET /metrics HTTP/1.1" 200 38504 "" "Prometheus/2.33.4" 2024-06-01T02:48:05.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:04 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:05.254 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:04 smithi146 ceph-mon[57708]: osdmap e131: 8 total, 8 up, 8 in 2024-06-01T02:48:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:04 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:05.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:04 smithi062 ceph-mon[64430]: osdmap e131: 8 total, 8 up, 8 in 2024-06-01T02:48:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:04 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:05.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:04 smithi062 ceph-mon[66884]: osdmap e131: 8 total, 8 up, 8 in 2024-06-01T02:48:06.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:05 smithi062 ceph-mon[64430]: pgmap v129: 161 pgs: 23 peering, 17 active+undersized+degraded, 28 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-06-01T02:48:06.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:05 smithi062 ceph-mon[66884]: pgmap v129: 161 pgs: 23 peering, 17 active+undersized+degraded, 28 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-06-01T02:48:06.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:05 smithi146 ceph-mon[57708]: pgmap v129: 161 pgs: 23 peering, 17 active+undersized+degraded, 28 active+undersized, 93 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 71/723 objects degraded (9.820%) 2024-06-01T02:48:07.263 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:06 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:48:06.986+0000 7f15439f1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: pgmap v130: 161 pgs: 23 peering, 12 active+undersized+degraded, 18 active+undersized, 108 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/723 objects degraded (5.809%) 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:48:08.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 42/723 objects degraded (5.809%), 12 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: pgmap v130: 161 pgs: 23 peering, 12 active+undersized+degraded, 18 active+undersized, 108 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/723 objects degraded (5.809%) 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:08.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-06-01T02:48:08.167 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:07 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 42/723 objects degraded (5.809%), 12 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: pgmap v130: 161 pgs: 23 peering, 12 active+undersized+degraded, 18 active+undersized, 108 active+clean; 585 KiB data, 185 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 42/723 objects degraded (5.809%) 2024-06-01T02:48:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-06-01T02:48:08.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:07 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 42/723 objects degraded (5.809%), 12 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:08.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:08 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/723 objects degraded (5.809%), 12 pgs degraded) 2024-06-01T02:48:08.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:08 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:48:08.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:48:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:48:08.659Z caller=alerting.go:353 level=warn component="rule manager" alert="unsupported value type" msg="Expanding alert template failed" err="error executing template __alert_CephOSDDown: template: __alert_CephOSDDown:1:358: executing \"__alert_CephOSDDown\" at : error calling query: found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" data="unsupported value type" 2024-06-01T02:48:08.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:48:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:48:08.661Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:48:09.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:08 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/723 objects degraded (5.809%), 12 pgs degraded) 2024-06-01T02:48:09.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:08 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:48:09.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:08 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 42/723 objects degraded (5.809%), 12 pgs degraded) 2024-06-01T02:48:09.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:08 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:48:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:09 smithi146 ceph-mon[57708]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-06-01T02:48:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:09 smithi062 ceph-mon[64430]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-06-01T02:48:10.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:09 smithi062 ceph-mon[66884]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 781 B/s rd, 0 op/s 2024-06-01T02:48:11.253 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:48:11.933 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:48:11.933 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (4m) 44s ago 9m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:48:11.933 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (4m) 9s ago 9m 51.4M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:48:11.933 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (4m) 44s ago 8m 54.1M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (3m) 9s ago 14m 453M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (4m) 44s ago 15m 507M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (3m) 44s ago 15m 50.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (2m) 9s ago 14m 35.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (3m) 44s ago 14m 37.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (4m) 44s ago 10m 18.2M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (4m) 9s ago 10m 15.5M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (2m) 44s ago 13m 53.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (104s) 44s ago 13m 44.4M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (83s) 44s ago 12m 42.0M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (49s) 44s ago 12m 11.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (14s) 9s ago 12m 11.7M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (11m) 9s ago 11m 58.1M 2992M 17.2.0 e1d6a67b021e a61294900cc6 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (11m) 9s ago 11m 56.8M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:48:11.934 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (11m) 9s ago 11m 61.6M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:48:11.935 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (4m) 9s ago 10m 75.1M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:48:11.935 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (8m) 44s ago 8m 76.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:48:11.935 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (8m) 9s ago 8m 75.8M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:48:11.935 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (8m) 44s ago 8m 75.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:48:11.935 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (8m) 9s ago 8m 74.6M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:48:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:11 smithi146 ceph-mon[57708]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-01T02:48:12.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:11 smithi146 ceph-mon[57708]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:11 smithi062 ceph-mon[64430]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-01T02:48:12.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:11 smithi062 ceph-mon[64430]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:12.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:11 smithi062 ceph-mon[66884]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-06-01T02:48:12.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:11 smithi062 ceph-mon[66884]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:12.352 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:48:12.352 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 5 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:12.353 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:48:12.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-06-01T02:48:12.354 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 10 2024-06-01T02:48:12.354 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:48:12.354 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "10/23 daemons upgraded", 2024-06-01T02:48:12.708 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:48:12.709 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:48:12.709 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:48:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[66884]: from='client.44314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[66884]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/19639570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:13.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:13.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[64430]: from='client.44314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:13.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[64430]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:13.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/19639570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:13.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:13.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:13.120 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:48:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:12 smithi146 ceph-mon[57708]: from='client.44314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:12 smithi146 ceph-mon[57708]: from='client.34392 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:12 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/19639570' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:13.885 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:48:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:48:13.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:48:14.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:13 smithi062 ceph-mon[64430]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:48:14.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:13 smithi062 ceph-mon[64430]: from='client.44332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:14.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:13 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/2053972983' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:48:14.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:13 smithi062 ceph-mon[66884]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:48:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:13 smithi062 ceph-mon[66884]: from='client.44332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:13 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/2053972983' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:48:14.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:13 smithi146 ceph-mon[57708]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:48:14.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:13 smithi146 ceph-mon[57708]: from='client.44332 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:14.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:13 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/2053972983' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:48:14.890 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:48:14] "GET /metrics HTTP/1.1" 200 38508 "" "Prometheus/2.33.4" 2024-06-01T02:48:15.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:14 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:15.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:14 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:15.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:14 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:16.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:15 smithi146 ceph-mon[57708]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s 2024-06-01T02:48:16.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:15 smithi062 ceph-mon[64430]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s 2024-06-01T02:48:16.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:15 smithi062 ceph-mon[66884]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1015 B/s rd, 0 op/s 2024-06-01T02:48:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:17 smithi146 ceph-mon[57708]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:17 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:18.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:17 smithi062 ceph-mon[64430]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:18.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:17 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:18.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:17 smithi062 ceph-mon[66884]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:18.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:17 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:18.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:48:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:48:18.659Z caller=manager.go:609 level=warn component="rule manager" group=osd msg="Evaluating rule failed" rule="alert: CephOSDFlapping\nexpr: (rate(ceph_osd_up[5m]) * on(ceph_daemon) group_left(hostname) ceph_osd_metadata)\n * 60 > 1\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.4.4\n severity: warning\n type: ceph_default\nannotations:\n description: OSD {{ $labels.ceph_daemon }} on {{ $labels.hostname }} was marked\n down and back up {{ $value | humanize }} times once a minute for 5 minutes. This\n may indicate a network issue (latency, packet loss, MTU mismatch) on the cluster\n network, or the public network if no cluster network is deployed. Check the network\n stats on the listed host(s).\n documentation: https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-osd#flapping-osds\n summary: Network issues are causing OSDs to flap (mark each other down)\n" err="found duplicate series for the match group {ceph_daemon=\"osd.0\"} on the right hand-side of the operation: [{__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"smithi062:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}, {__name__=\"ceph_osd_metadata\", ceph_daemon=\"osd.0\", ceph_version=\"ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)\", cluster_addr=\"172.21.15.62\", device_class=\"ssd\", hostname=\"smithi062\", instance=\"172.21.15.146:9283\", job=\"ceph\", objectstore=\"bluestore\", public_addr=\"172.21.15.62\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:48:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:19 smithi146 ceph-mon[57708]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:48:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:19 smithi062 ceph-mon[64430]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:48:20.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:19 smithi062 ceph-mon[66884]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:48:22.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:21 smithi146 ceph-mon[57708]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:22.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:21 smithi062 ceph-mon[64430]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:21 smithi062 ceph-mon[66884]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:23.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:22 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:23.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:22 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:23.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:22 smithi146 ceph-mon[57708]: Upgrade: osd.5 is safe to restart 2024-06-01T02:48:23.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:22 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:23.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:22 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:23.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:22 smithi062 ceph-mon[64430]: Upgrade: osd.5 is safe to restart 2024-06-01T02:48:23.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:22 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:23.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:22 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-06-01T02:48:23.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:22 smithi062 ceph-mon[66884]: Upgrade: osd.5 is safe to restart 2024-06-01T02:48:23.956 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:48:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:48:23.505Z caller=manager.go:609 level=warn component="rule manager" group=nodes msg="Evaluating rule failed" rule="alert: CephNodeDiskspaceWarning\nexpr: predict_linear(node_filesystem_free_bytes{device=~\"/.*\"}[2d], 3600 * 24 * 5)\n * on(instance) group_left(nodename) node_uname_info < 0\nlabels:\n oid: 1.3.6.1.4.1.50495.1.2.1.8.4\n severity: warning\n type: ceph_default\nannotations:\n description: Mountpoint {{ $labels.mountpoint }} on {{ $labels.nodename }} will\n be full in less than 5 days based on the 48 hour trailing fill rate.\n summary: Host filesystem free space is getting low\n" err="found duplicate series for the match group {instance=\"smithi062\"} on the right hand-side of the operation: [{__name__=\"node_uname_info\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}, {__name__=\"node_uname_info\", cluster=\"176c65a6-1fbf-11ef-bc9b-c7b262605968\", domainname=\"(none)\", instance=\"smithi062\", job=\"node\", machine=\"x86_64\", nodename=\"smithi062\", release=\"5.14.0-452.el9.x86_64\", sysname=\"Linux\", version=\"#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024\"}];many-to-many matching not allowed: matching labels must be unique on one side" 2024-06-01T02:48:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:24 smithi146 ceph-mon[57708]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:48:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:24 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:24 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:48:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:24 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:24.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[64430]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:48:24.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:24.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:48:24.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:24.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[66884]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:48:24.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:24.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-01T02:48:24.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:25.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:48:24] "GET /metrics HTTP/1.1" 200 38508 "" "Prometheus/2.33.4" 2024-06-01T02:48:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:25 smithi146 ceph-mon[57708]: Upgrade: Updating osd.5 2024-06-01T02:48:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:25 smithi146 ceph-mon[57708]: Deploying daemon osd.5 on smithi146 2024-06-01T02:48:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:25 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:25 smithi062 ceph-mon[64430]: Upgrade: Updating osd.5 2024-06-01T02:48:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:25 smithi062 ceph-mon[64430]: Deploying daemon osd.5 on smithi146 2024-06-01T02:48:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:25 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:25 smithi062 ceph-mon[66884]: Upgrade: Updating osd.5 2024-06-01T02:48:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:25 smithi062 ceph-mon[66884]: Deploying daemon osd.5 on smithi146 2024-06-01T02:48:25.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:25 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:26.455 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:26 smithi146 ceph-mon[57708]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:26 smithi062 ceph-mon[64430]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:26.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:26 smithi062 ceph-mon[66884]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:27.118 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:26 smithi146 systemd[1]: Stopping Ceph osd.5 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:48:27.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:27 smithi146 ceph-mon[57708]: osd.5 marked itself down and dead 2024-06-01T02:48:27.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[32224]: 2024-06-01T02:48:27.117+0000 7f6c53fd5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:48:27.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[32224]: 2024-06-01T02:48:27.117+0000 7f6c53fd5700 -1 osd.5 131 *** Got signal Terminated *** 2024-06-01T02:48:27.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[32224]: 2024-06-01T02:48:27.117+0000 7f6c53fd5700 -1 osd.5 131 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:48:27.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:27 smithi062 ceph-mon[64430]: osd.5 marked itself down and dead 2024-06-01T02:48:27.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:27 smithi062 ceph-mon[66884]: osd.5 marked itself down and dead 2024-06-01T02:48:27.953 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:27 smithi146 podman[62711]: 2024-06-01 02:48:27.66965885 +0000 UTC m=+0.678442885 container died a61294900cc6eb93aa72a0d3d234a832988b1a36b2317112be3f915efd182f32 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, release=754, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, version=8, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, architecture=x86_64, RELEASE=HEAD, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8) 2024-06-01T02:48:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:28 smithi146 ceph-mon[57708]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:28 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:48:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:28 smithi146 ceph-mon[57708]: osdmap e132: 8 total, 7 up, 8 in 2024-06-01T02:48:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:28.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:28 smithi146 podman[62711]: 2024-06-01 02:48:28.117837691 +0000 UTC m=+1.126621758 container cleanup a61294900cc6eb93aa72a0d3d234a832988b1a36b2317112be3f915efd182f32 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8) 2024-06-01T02:48:28.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:28 smithi146 bash[62711]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5 2024-06-01T02:48:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[66884]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:48:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[66884]: osdmap e132: 8 total, 7 up, 8 in 2024-06-01T02:48:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:28.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:28.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[64430]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:28.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:48:28.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[64430]: osdmap e132: 8 total, 7 up, 8 in 2024-06-01T02:48:28.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:28.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:28.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:28 smithi146 podman[62725]: 2024-06-01 02:48:28.336207432 +0000 UTC m=+0.660750611 container remove a61294900cc6eb93aa72a0d3d234a832988b1a36b2317112be3f915efd182f32 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_BRANCH=HEAD, version=8, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, RELEASE=HEAD) 2024-06-01T02:48:29.047 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:28 smithi146 podman[62816]: 2024-06-01 02:48:28.690858946 +0000 UTC m=+0.017072689 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:48:29.047 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:28 smithi146 podman[62816]: 2024-06-01 02:48:28.837180262 +0000 UTC m=+0.163393986 container create 1b50ef07b8ab52c18368466f5bf9bf9a6e657393467d95cf9e700d9f1344d9f3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-01T02:48:29.357 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:29 smithi146 ceph-mon[57708]: osdmap e133: 8 total, 7 up, 8 in 2024-06-01T02:48:29.357 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:29 smithi146 podman[62816]: 2024-06-01 02:48:29.047682624 +0000 UTC m=+0.373896333 container init 1b50ef07b8ab52c18368466f5bf9bf9a6e657393467d95cf9e700d9f1344d9f3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-06-01T02:48:29.357 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:29 smithi146 podman[62816]: 2024-06-01 02:48:29.051092692 +0000 UTC m=+0.377306412 container start 1b50ef07b8ab52c18368466f5bf9bf9a6e657393467d95cf9e700d9f1344d9f3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:48:29.357 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:29 smithi146 podman[62816]: 2024-06-01 02:48:29.139296167 +0000 UTC m=+0.465509902 container attach 1b50ef07b8ab52c18368466f5bf9bf9a6e657393467d95cf9e700d9f1344d9f3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-06-01T02:48:29.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:29 smithi062 ceph-mon[64430]: osdmap e133: 8 total, 7 up, 8 in 2024-06-01T02:48:29.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:29 smithi062 ceph-mon[66884]: osdmap e133: 8 total, 7 up, 8 in 2024-06-01T02:48:29.706 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:29 smithi146 podman[62816]: 2024-06-01 02:48:29.358076675 +0000 UTC m=+0.684290397 container died 1b50ef07b8ab52c18368466f5bf9bf9a6e657393467d95cf9e700d9f1344d9f3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0) 2024-06-01T02:48:30.198 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 podman[62842]: 2024-06-01 02:48:30.009665599 +0000 UTC m=+0.641705294 container remove 1b50ef07b8ab52c18368466f5bf9bf9a6e657393467d95cf9e700d9f1344d9f3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-01T02:48:30.199 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.5.service: Deactivated successfully. 2024-06-01T02:48:30.199 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 systemd[1]: Stopped Ceph osd.5 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:48:30.199 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.5.service: Consumed 8.604s CPU time. 2024-06-01T02:48:30.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:30 smithi146 ceph-mon[57708]: pgmap v143: 161 pgs: 6 active+undersized+degraded, 5 peering, 15 stale+active+clean, 9 active+undersized, 126 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 19/723 objects degraded (2.628%) 2024-06-01T02:48:30.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:30 smithi146 ceph-mon[57708]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-01T02:48:30.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:30 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 19/723 objects degraded (2.628%), 6 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:30.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 systemd[1]: Starting Ceph osd.5 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:48:30.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:30 smithi062 ceph-mon[64430]: pgmap v143: 161 pgs: 6 active+undersized+degraded, 5 peering, 15 stale+active+clean, 9 active+undersized, 126 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 19/723 objects degraded (2.628%) 2024-06-01T02:48:30.566 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:30 smithi062 ceph-mon[64430]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-01T02:48:30.566 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:30 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 19/723 objects degraded (2.628%), 6 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:30.567 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:30 smithi062 ceph-mon[66884]: pgmap v143: 161 pgs: 6 active+undersized+degraded, 5 peering, 15 stale+active+clean, 9 active+undersized, 126 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 19/723 objects degraded (2.628%) 2024-06-01T02:48:30.567 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:30 smithi062 ceph-mon[66884]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-06-01T02:48:30.567 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:30 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 19/723 objects degraded (2.628%), 6 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:30.779 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 podman[62940]: 2024-06-01 02:48:30.368320056 +0000 UTC m=+0.017985889 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:48:30.779 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 podman[62940]: 2024-06-01 02:48:30.469082962 +0000 UTC m=+0.118748832 container create ae53cc1277c7ed2585c752502f0bb1c67e6adb17a5953f4ff8af635c737a57f7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-06-01T02:48:30.779 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 podman[62940]: 2024-06-01 02:48:30.696753321 +0000 UTC m=+0.346419155 container init ae53cc1277c7ed2585c752502f0bb1c67e6adb17a5953f4ff8af635c737a57f7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-01T02:48:30.780 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 podman[62940]: 2024-06-01 02:48:30.700076327 +0000 UTC m=+0.349742153 container start ae53cc1277c7ed2585c752502f0bb1c67e6adb17a5953f4ff8af635c737a57f7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-06-01T02:48:30.780 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:30 smithi146 podman[62940]: 2024-06-01 02:48:30.779919546 +0000 UTC m=+0.429585366 container attach ae53cc1277c7ed2585c752502f0bb1c67e6adb17a5953f4ff8af635c737a57f7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528) 2024-06-01T02:48:31.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-01T02:48:31.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-01T02:48:31.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-01T02:48:31.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-06-01T02:48:31.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 bash[62940]: --> ceph-volume raw activate successful for osd ID: 5 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate[62954]: --> ceph-volume raw activate successful for osd ID: 5 2024-06-01T02:48:31.457 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 podman[62940]: 2024-06-01 02:48:31.172193068 +0000 UTC m=+0.821858894 container died ae53cc1277c7ed2585c752502f0bb1c67e6adb17a5953f4ff8af635c737a57f7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-01T02:48:32.203 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:31 smithi146 podman[63070]: 2024-06-01 02:48:31.875528113 +0000 UTC m=+0.693841251 container remove ae53cc1277c7ed2585c752502f0bb1c67e6adb17a5953f4ff8af635c737a57f7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-activate, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-01T02:48:32.203 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 podman[63129]: 2024-06-01 02:48:31.998581655 +0000 UTC m=+0.018123498 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:48:32.203 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 podman[63129]: 2024-06-01 02:48:32.11809346 +0000 UTC m=+0.137635298 container create 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-01T02:48:32.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:32 smithi146 ceph-mon[57708]: pgmap v144: 161 pgs: 6 active+undersized+degraded, 5 peering, 15 stale+active+clean, 9 active+undersized, 126 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 19/723 objects degraded (2.628%) 2024-06-01T02:48:32.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 podman[63129]: 2024-06-01 02:48:32.34544951 +0000 UTC m=+0.364991319 container init 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-06-01T02:48:32.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 podman[63129]: 2024-06-01 02:48:32.348631921 +0000 UTC m=+0.368173729 container start 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:48:32.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 ceph-osd[63148]: -- 172.21.15.146:0/2959961890 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc16117680 con 0x55cc160e0c00 2024-06-01T02:48:32.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:32 smithi062 ceph-mon[64430]: pgmap v144: 161 pgs: 6 active+undersized+degraded, 5 peering, 15 stale+active+clean, 9 active+undersized, 126 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 19/723 objects degraded (2.628%) 2024-06-01T02:48:32.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:32 smithi062 ceph-mon[66884]: pgmap v144: 161 pgs: 6 active+undersized+degraded, 5 peering, 15 stale+active+clean, 9 active+undersized, 126 active+clean; 585 KiB data, 193 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 19/723 objects degraded (2.628%) 2024-06-01T02:48:32.902 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 bash[63129]: 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d 2024-06-01T02:48:32.903 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 systemd[1]: Started Ceph osd.5 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:48:33.206 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:48:32.902+0000 7f2390b1a7c0 -1 Falling back to public interface 2024-06-01T02:48:33.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:33.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:33.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:33 smithi146 ceph-mon[57708]: pgmap v145: 161 pgs: 20 active+undersized+degraded, 5 peering, 34 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:33.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:33.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:33.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:33 smithi062 ceph-mon[64430]: pgmap v145: 161 pgs: 20 active+undersized+degraded, 5 peering, 34 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:33.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:33.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:33.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:33 smithi062 ceph-mon[66884]: pgmap v145: 161 pgs: 20 active+undersized+degraded, 5 peering, 34 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:34.828 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:34 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:34 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:35.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:34 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:35.014 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:48:34] "GET /metrics HTTP/1.1" 200 38512 "" "Prometheus/2.33.4" 2024-06-01T02:48:35.833 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:35 smithi146 ceph-mon[57708]: pgmap v146: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:35.833 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:48:35.615+0000 7f2390b1a7c0 -1 osd.5 0 read_superblock omap replica is missing. 2024-06-01T02:48:35.833 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:48:35.645+0000 7f2390b1a7c0 -1 osd.5 131 log_to_monitors true 2024-06-01T02:48:36.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:35 smithi062 ceph-mon[64430]: pgmap v146: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:36.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:35 smithi062 ceph-mon[66884]: pgmap v146: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:36.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:36 smithi146 ceph-mon[57708]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-01T02:48:36.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:36 smithi146 ceph-mon[57708]: from='osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:48:36.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:36 smithi146 ceph-mon[57708]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:48:37.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:36 smithi062 ceph-mon[64430]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-01T02:48:37.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:36 smithi062 ceph-mon[64430]: from='osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:48:37.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:36 smithi062 ceph-mon[64430]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:48:37.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:36 smithi062 ceph-mon[66884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-06-01T02:48:37.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:36 smithi062 ceph-mon[66884]: from='osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:48:37.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:36 smithi062 ceph-mon[66884]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-01T02:48:37.206 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:48:37.021+0000 7f237df69700 -1 osd.5 131 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:48:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T02:48:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: osdmap e134: 8 total, 7 up, 8 in 2024-06-01T02:48:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: from='osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:37.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:37.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: pgmap v148: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:37.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:37.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:37.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: from='osd.5 ' entity='osd.5' 2024-06-01T02:48:37.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:37 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 20 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T02:48:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: osdmap e134: 8 total, 7 up, 8 in 2024-06-01T02:48:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: from='osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: pgmap v148: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: from='osd.5 ' entity='osd.5' 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 20 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: osdmap e134: 8 total, 7 up, 8 in 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: from='osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: pgmap v148: 161 pgs: 20 active+undersized+degraded, 39 active+undersized, 102 active+clean; 585 KiB data, 198 MiB used, 715 GiB / 715 GiB avail; 81/723 objects degraded (11.203%) 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: from='osd.5 ' entity='osd.5' 2024-06-01T02:48:38.015 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:37 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 81/723 objects degraded (11.203%), 20 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:38.807 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:38 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:48:38.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:38 smithi146 ceph-mon[57708]: osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548] boot 2024-06-01T02:48:38.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:38 smithi146 ceph-mon[57708]: osdmap e135: 8 total, 8 up, 8 in 2024-06-01T02:48:38.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:38 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:48:38.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:38 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.808 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:38 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[64430]: osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548] boot 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[64430]: osdmap e135: 8 total, 8 up, 8 in 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[66884]: osd.5 [v2:172.21.15.146:6808/4294956548,v1:172.21.15.146:6809/4294956548] boot 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[66884]: osdmap e135: 8 total, 8 up, 8 in 2024-06-01T02:48:38.915 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-01T02:48:38.916 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:38.916 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:38 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:40.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:40 smithi146 ceph-mon[57708]: pgmap v150: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 69/723 objects degraded (9.544%) 2024-06-01T02:48:40.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:40 smithi146 ceph-mon[57708]: osdmap e136: 8 total, 8 up, 8 in 2024-06-01T02:48:40.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:40 smithi062 ceph-mon[64430]: pgmap v150: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 69/723 objects degraded (9.544%) 2024-06-01T02:48:40.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:40 smithi062 ceph-mon[64430]: osdmap e136: 8 total, 8 up, 8 in 2024-06-01T02:48:40.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:40 smithi062 ceph-mon[66884]: pgmap v150: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 69/723 objects degraded (9.544%) 2024-06-01T02:48:40.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:40 smithi062 ceph-mon[66884]: osdmap e136: 8 total, 8 up, 8 in 2024-06-01T02:48:42.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:41 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:48:41.637+0000 7f15439f1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:48:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: pgmap v152: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/723 objects degraded (9.544%) 2024-06-01T02:48:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:42.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:48:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:48:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:42 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:42.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: pgmap v152: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/723 objects degraded (9.544%) 2024-06-01T02:48:42.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: pgmap v152: 161 pgs: 10 peering, 16 active+undersized+degraded, 33 active+undersized, 102 active+clean; 585 KiB data, 213 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 69/723 objects degraded (9.544%) 2024-06-01T02:48:42.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:42.515 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:42 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:43 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:43 smithi146 ceph-mon[57708]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:48:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:43 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 69/723 objects degraded (9.544%), 16 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:43.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:43.497 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:43.497 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[64430]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:48:43.497 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 69/723 objects degraded (9.544%), 16 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[66884]: Upgrade: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 69/723 objects degraded (9.544%), 16 pgs degraded (PG_DEGRADED) 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:43.498 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:43.499 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (5m) 76s ago 10m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (5m) 7s ago 9m 51.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (4m) 76s ago 9m 54.1M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (4m) 7s ago 14m 453M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (5m) 76s ago 16m 507M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (4m) 76s ago 16m 50.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (3m) 7s ago 14m 36.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (3m) 76s ago 15m 37.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (5m) 76s ago 10m 18.2M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (5m) 7s ago 10m 15.1M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:48:44.203 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (2m) 76s ago 14m 53.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (2m) 76s ago 13m 44.4M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (116s) 76s ago 13m 42.0M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (81s) 76s ago 13m 11.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (46s) 7s ago 12m 50.8M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (12s) 7s ago 12m 11.6M 2992M 19.0.0-2946-g3d89accf b045ac545f82 31b032e2fbe8 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (11m) 7s ago 11m 58.6M 2992M 17.2.0 e1d6a67b021e a1c894465c5a 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (11m) 7s ago 11m 62.3M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (4m) 7s ago 10m 75.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (9m) 76s ago 9m 76.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (9m) 7s ago 9m 76.1M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (9m) 76s ago 9m 75.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:48:44.204 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (9m) 7s ago 9m 74.8M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:48:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:44 smithi146 ceph-mon[57708]: pgmap v153: 161 pgs: 10 peering, 4 active+undersized, 147 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:48:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:44 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/723 objects degraded (9.544%), 16 pgs degraded) 2024-06-01T02:48:44.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:44 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:48:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-mon[64430]: pgmap v153: 161 pgs: 10 peering, 4 active+undersized, 147 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:48:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/723 objects degraded (9.544%), 16 pgs degraded) 2024-06-01T02:48:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:48:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-mon[66884]: pgmap v153: 161 pgs: 10 peering, 4 active+undersized, 147 active+clean; 585 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:48:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 69/723 objects degraded (9.544%), 16 pgs degraded) 2024-06-01T02:48:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:48:44.619 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:48:44.619 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:48:44.619 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:48:44.619 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:44.619 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 6 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 11 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:48:44.620 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:48:44.968 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:48:44.968 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:48:44.968 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:48:44.968 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:48:44.968 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:48:44.968 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:48:44.969 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:48:44.969 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:48:44.969 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "11/23 daemons upgraded", 2024-06-01T02:48:44.969 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:48:44.969 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:48:44.969 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:48:44.969 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:48:44] "GET /metrics HTTP/1.1" 200 38520 "" "Prometheus/2.33.4" 2024-06-01T02:48:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[66884]: from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[66884]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[66884]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/665500482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:45.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[64430]: from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:45.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[64430]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[64430]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:45 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/665500482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:45.385 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:48:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:45 smithi146 ceph-mon[57708]: from='client.44350 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:45 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:45 smithi146 ceph-mon[57708]: from='client.44356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:45 smithi146 ceph-mon[57708]: from='client.34434 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:45 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/665500482' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:48:46.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:46 smithi146 ceph-mon[57708]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:48:46.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:46 smithi146 ceph-mon[57708]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:46.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:46 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/444748998' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:48:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:46 smithi062 ceph-mon[64430]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:48:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:46 smithi062 ceph-mon[64430]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:46 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/444748998' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:48:46.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:46 smithi062 ceph-mon[66884]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:48:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:46 smithi062 ceph-mon[66884]: from='client.34446 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:48:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:46 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/444748998' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:48:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:48 smithi146 ceph-mon[57708]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 359 B/s rd, 0 op/s 2024-06-01T02:48:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:48 smithi062 ceph-mon[64430]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 359 B/s rd, 0 op/s 2024-06-01T02:48:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:48 smithi062 ceph-mon[66884]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 359 B/s rd, 0 op/s 2024-06-01T02:48:50.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:50 smithi146 ceph-mon[57708]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-06-01T02:48:50.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:50 smithi062 ceph-mon[64430]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-06-01T02:48:50.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:50 smithi062 ceph-mon[66884]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 818 B/s rd, 0 op/s 2024-06-01T02:48:52.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:52 smithi146 ceph-mon[57708]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 0 op/s 2024-06-01T02:48:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:52 smithi062 ceph-mon[64430]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 0 op/s 2024-06-01T02:48:52.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:52 smithi062 ceph-mon[66884]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 709 B/s rd, 0 op/s 2024-06-01T02:48:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:53 smithi146 ceph-mon[57708]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:48:53.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:54.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:53 smithi062 ceph-mon[64430]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:48:54.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:54.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:53 smithi062 ceph-mon[66884]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:48:54.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:54.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:54 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:55.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:54 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:55.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:54 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:48:55.014 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:48:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:48:54] "GET /metrics HTTP/1.1" 200 38520 "" "Prometheus/2.33.4" 2024-06-01T02:48:55.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:55 smithi146 ceph-mon[57708]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:56.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:55 smithi062 ceph-mon[64430]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:56.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:55 smithi062 ceph-mon[66884]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:56.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:56 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:57.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:56 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:57.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:56 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:57.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:57 smithi146 ceph-mon[57708]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:57.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:57 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:57.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:57 smithi146 ceph-mon[57708]: Upgrade: osd.6 is safe to restart 2024-06-01T02:48:57.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:57.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:58.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[64430]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:58.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:58.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[64430]: Upgrade: osd.6 is safe to restart 2024-06-01T02:48:58.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:58.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:58.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[66884]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:48:58.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-06-01T02:48:58.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[66884]: Upgrade: osd.6 is safe to restart 2024-06-01T02:48:58.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:58.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:48:59.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:59 smithi146 ceph-mon[57708]: Upgrade: Updating osd.6 2024-06-01T02:48:59.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:59.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:48:59.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:59.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:48:59 smithi146 ceph-mon[57708]: Deploying daemon osd.6 on smithi146 2024-06-01T02:48:59.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[64430]: Upgrade: Updating osd.6 2024-06-01T02:48:59.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:59.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:48:59.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:59.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[64430]: Deploying daemon osd.6 on smithi146 2024-06-01T02:48:59.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[66884]: Upgrade: Updating osd.6 2024-06-01T02:48:59.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:48:59.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-06-01T02:48:59.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:48:59.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:48:59 smithi062 ceph-mon[66884]: Deploying daemon osd.6 on smithi146 2024-06-01T02:49:00.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:00 smithi146 ceph-mon[57708]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:49:00.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:00 smithi062 ceph-mon[64430]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:49:00.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:00 smithi062 ceph-mon[66884]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:49:01.666 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:01 smithi146 systemd[1]: Stopping Ceph osd.6 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:49:01.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[35104]: 2024-06-01T02:49:01.665+0000 7fe3398cd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:49:01.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[35104]: 2024-06-01T02:49:01.665+0000 7fe3398cd700 -1 osd.6 136 *** Got signal Terminated *** 2024-06-01T02:49:01.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:01 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[35104]: 2024-06-01T02:49:01.665+0000 7fe3398cd700 -1 osd.6 136 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:49:02.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:02 smithi062 ceph-mon[64430]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:49:02.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:02 smithi062 ceph-mon[64430]: osd.6 marked itself down and dead 2024-06-01T02:49:02.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:02 smithi062 ceph-mon[66884]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:49:02.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:02 smithi062 ceph-mon[66884]: osd.6 marked itself down and dead 2024-06-01T02:49:02.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:02 smithi146 ceph-mon[57708]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:49:02.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:02 smithi146 ceph-mon[57708]: osd.6 marked itself down and dead 2024-06-01T02:49:03.066 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:02 smithi146 podman[64997]: 2024-06-01 02:49:02.791104816 +0000 UTC m=+1.240766834 container died a1c894465c5a8b755a0449a08d14fb4442e1b7a4864b2979d99e8bb2618bab56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, distribution-scope=public, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vendor=Red Hat, Inc., name=centos-stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, version=8) 2024-06-01T02:49:03.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:03 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:49:03.416 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:03 smithi146 ceph-mon[57708]: osdmap e137: 8 total, 7 up, 8 in 2024-06-01T02:49:03.417 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:03 smithi146 podman[64997]: 2024-06-01 02:49:03.173965328 +0000 UTC m=+1.623627329 container cleanup a1c894465c5a8b755a0449a08d14fb4442e1b7a4864b2979d99e8bb2618bab56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, ceph=True, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.component=centos-stream-container, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, vcs-type=git) 2024-06-01T02:49:03.417 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:03 smithi146 bash[64997]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6 2024-06-01T02:49:03.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:03 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:49:03.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:03 smithi062 ceph-mon[64430]: osdmap e137: 8 total, 7 up, 8 in 2024-06-01T02:49:03.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:03 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:49:03.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:03 smithi062 ceph-mon[66884]: osdmap e137: 8 total, 7 up, 8 in 2024-06-01T02:49:03.706 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:03 smithi146 podman[65013]: 2024-06-01 02:49:03.417020984 +0000 UTC m=+0.627758368 container remove a1c894465c5a8b755a0449a08d14fb4442e1b7a4864b2979d99e8bb2618bab56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, distribution-scope=public, io.buildah.version=1.19.8, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64) 2024-06-01T02:49:04.082 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:03 smithi146 podman[65102]: 2024-06-01 02:49:03.66550846 +0000 UTC m=+0.017544539 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:49:04.082 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:03 smithi146 podman[65102]: 2024-06-01 02:49:03.809684824 +0000 UTC m=+0.161720921 container create c8d870e2e8629e99a9ab02bf25eea66bc41987a8bad1e7e05e419cd450e91500 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux ) 2024-06-01T02:49:04.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:04 smithi146 ceph-mon[57708]: pgmap v164: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:49:04.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:04 smithi146 ceph-mon[57708]: osdmap e138: 8 total, 7 up, 8 in 2024-06-01T02:49:04.385 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 podman[65102]: 2024-06-01 02:49:04.082539657 +0000 UTC m=+0.434575725 container init c8d870e2e8629e99a9ab02bf25eea66bc41987a8bad1e7e05e419cd450e91500 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-01T02:49:04.385 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 podman[65102]: 2024-06-01 02:49:04.086034235 +0000 UTC m=+0.438070295 container start c8d870e2e8629e99a9ab02bf25eea66bc41987a8bad1e7e05e419cd450e91500 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-01T02:49:04.385 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 podman[65102]: 2024-06-01 02:49:04.270145382 +0000 UTC m=+0.622181450 container attach c8d870e2e8629e99a9ab02bf25eea66bc41987a8bad1e7e05e419cd450e91500 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-01T02:49:04.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:04 smithi062 ceph-mon[64430]: pgmap v164: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:49:04.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:04 smithi062 ceph-mon[64430]: osdmap e138: 8 total, 7 up, 8 in 2024-06-01T02:49:04.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:04 smithi062 ceph-mon[66884]: pgmap v164: 161 pgs: 13 stale+active+clean, 148 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:49:04.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:04 smithi062 ceph-mon[66884]: osdmap e138: 8 total, 7 up, 8 in 2024-06-01T02:49:04.698 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 podman[65102]: 2024-06-01 02:49:04.385528457 +0000 UTC m=+0.737564522 container died c8d870e2e8629e99a9ab02bf25eea66bc41987a8bad1e7e05e419cd450e91500 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-01T02:49:05.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:49:04] "GET /metrics HTTP/1.1" 200 38519 "" "Prometheus/2.33.4" 2024-06-01T02:49:05.260 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 podman[65128]: 2024-06-01 02:49:04.965142009 +0000 UTC m=+0.573880510 container remove c8d870e2e8629e99a9ab02bf25eea66bc41987a8bad1e7e05e419cd450e91500 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-01T02:49:05.260 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.6.service: Deactivated successfully. 2024-06-01T02:49:05.260 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 systemd[1]: Stopped Ceph osd.6 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:49:05.260 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:04 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.6.service: Consumed 8.117s CPU time. 2024-06-01T02:49:05.260 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:05 smithi146 systemd[1]: Starting Ceph osd.6 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:49:05.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:05 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:05.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:05 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:05.532 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:05 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:05.532 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:05 smithi146 podman[65227]: 2024-06-01 02:49:05.409957878 +0000 UTC m=+0.110028253 container create ceda4a5cc0b4aaee35fbb3457e8e283c2dc50680a66b90ea68315eca74c4a486 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-01T02:49:05.533 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:05 smithi146 podman[65227]: 2024-06-01 02:49:05.317068062 +0000 UTC m=+0.017138413 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:49:05.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:05 smithi146 podman[65227]: 2024-06-01 02:49:05.63734583 +0000 UTC m=+0.337416173 container init ceda4a5cc0b4aaee35fbb3457e8e283c2dc50680a66b90ea68315eca74c4a486 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-01T02:49:05.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:05 smithi146 podman[65227]: 2024-06-01 02:49:05.641128694 +0000 UTC m=+0.341199025 container start ceda4a5cc0b4aaee35fbb3457e8e283c2dc50680a66b90ea68315eca74c4a486 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc) 2024-06-01T02:49:05.956 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:05 smithi146 podman[65227]: 2024-06-01 02:49:05.737212319 +0000 UTC m=+0.437282666 container attach ceda4a5cc0b4aaee35fbb3457e8e283c2dc50680a66b90ea68315eca74c4a486 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-01T02:49:06.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-mon[57708]: pgmap v166: 161 pgs: 4 active+undersized+degraded, 12 stale+active+clean, 6 active+undersized, 139 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17/723 objects degraded (2.351%) 2024-06-01T02:49:06.424 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 17/723 objects degraded (2.351%), 4 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:06.424 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-01T02:49:06.424 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-01T02:49:06.424 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-01T02:49:06.424 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 bash[65227]: --> ceph-volume raw activate successful for osd ID: 6 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate[65242]: --> ceph-volume raw activate successful for osd ID: 6 2024-06-01T02:49:06.425 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 podman[65227]: 2024-06-01 02:49:06.120609088 +0000 UTC m=+0.820679427 container died ceda4a5cc0b4aaee35fbb3457e8e283c2dc50680a66b90ea68315eca74c4a486 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:49:06.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:06 smithi062 ceph-mon[64430]: pgmap v166: 161 pgs: 4 active+undersized+degraded, 12 stale+active+clean, 6 active+undersized, 139 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17/723 objects degraded (2.351%) 2024-06-01T02:49:06.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:06 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 17/723 objects degraded (2.351%), 4 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:06.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:06 smithi062 ceph-mon[66884]: pgmap v166: 161 pgs: 4 active+undersized+degraded, 12 stale+active+clean, 6 active+undersized, 139 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17/723 objects degraded (2.351%) 2024-06-01T02:49:06.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:06 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 17/723 objects degraded (2.351%), 4 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:06.974 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 podman[65374]: 2024-06-01 02:49:06.724250808 +0000 UTC m=+0.592320293 container remove ceda4a5cc0b4aaee35fbb3457e8e283c2dc50680a66b90ea68315eca74c4a486 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-06-01T02:49:06.974 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 podman[65434]: 2024-06-01 02:49:06.839308291 +0000 UTC m=+0.017524955 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:49:06.975 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:06 smithi146 podman[65434]: 2024-06-01 02:49:06.975068039 +0000 UTC m=+0.153284693 container create 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-01T02:49:07.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:07 smithi146 podman[65434]: 2024-06-01 02:49:07.186227805 +0000 UTC m=+0.364444477 container init 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True) 2024-06-01T02:49:07.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:07 smithi146 podman[65434]: 2024-06-01 02:49:07.190472443 +0000 UTC m=+0.368689107 container start 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-06-01T02:49:07.252 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:07 smithi146 ceph-osd[65453]: -- 172.21.15.146:0/1585192810 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563963b6d680 con 0x563963b37000 2024-06-01T02:49:07.706 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:07 smithi146 bash[65434]: 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 2024-06-01T02:49:07.706 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:07 smithi146 systemd[1]: Started Ceph osd.6 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:49:08.099 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:07 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:49:07.735+0000 7efc941757c0 -1 Falling back to public interface 2024-06-01T02:49:08.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:08 smithi146 ceph-mon[57708]: pgmap v167: 161 pgs: 6 active+undersized+degraded, 9 stale+active+clean, 12 active+undersized, 134 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 30/723 objects degraded (4.149%) 2024-06-01T02:49:08.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:08 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:08.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:08 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:08.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:08 smithi062 ceph-mon[64430]: pgmap v167: 161 pgs: 6 active+undersized+degraded, 9 stale+active+clean, 12 active+undersized, 134 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 30/723 objects degraded (4.149%) 2024-06-01T02:49:08.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:08 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:08.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:08 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:08.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:08 smithi062 ceph-mon[66884]: pgmap v167: 161 pgs: 6 active+undersized+degraded, 9 stale+active+clean, 12 active+undersized, 134 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 383 B/s rd, 0 op/s; 30/723 objects degraded (4.149%) 2024-06-01T02:49:08.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:08 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:08.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:08 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:10.600 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:10 smithi146 ceph-mon[57708]: pgmap v168: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:10.600 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:49:10.437+0000 7efc941757c0 -1 osd.6 0 read_superblock omap replica is missing. 2024-06-01T02:49:10.600 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:49:10.471+0000 7efc941757c0 -1 osd.6 136 log_to_monitors true 2024-06-01T02:49:10.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:10 smithi062 ceph-mon[64430]: pgmap v168: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:10.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:10 smithi062 ceph-mon[66884]: pgmap v168: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:11.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:11 smithi146 ceph-mon[57708]: from='osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:49:11.450 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:11 smithi146 ceph-mon[57708]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:49:11.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:11 smithi062 ceph-mon[64430]: from='osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:49:11.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:11 smithi062 ceph-mon[64430]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:49:11.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:11 smithi062 ceph-mon[66884]: from='osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:49:11.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:11 smithi062 ceph-mon[66884]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-06-01T02:49:12.206 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:49:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:49:11.754+0000 7efc815c4700 -1 osd.6 136 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:49:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: pgmap v169: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-01T02:49:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: from='osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: osdmap e139: 8 total, 7 up, 8 in 2024-06-01T02:49:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: from='osd.6 ' entity='osd.6' 2024-06-01T02:49:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:12.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:12 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 84/723 objects degraded (11.618%), 18 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:12.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: pgmap v169: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: from='osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: osdmap e139: 8 total, 7 up, 8 in 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: from='osd.6 ' entity='osd.6' 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 84/723 objects degraded (11.618%), 18 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: pgmap v169: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 218 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-06-01T02:49:12.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: from='osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:12.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: osdmap e139: 8 total, 7 up, 8 in 2024-06-01T02:49:12.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:12.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: from='osd.6 ' entity='osd.6' 2024-06-01T02:49:12.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:12.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:12.765 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:12 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 84/723 objects degraded (11.618%), 18 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:13.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:13.790 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: pgmap v171: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310] boot 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: osdmap e140: 8 total, 8 up, 8 in 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:13.791 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:14.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:14.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: pgmap v171: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:14.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310] boot 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: osdmap e140: 8 total, 8 up, 8 in 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: pgmap v171: 161 pgs: 18 active+undersized+degraded, 35 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 84/723 objects degraded (11.618%) 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: osd.6 [v2:172.21.15.146:6816/1324066310,v1:172.21.15.146:6817/1324066310] boot 2024-06-01T02:49:14.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: osdmap e140: 8 total, 8 up, 8 in 2024-06-01T02:49:14.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-06-01T02:49:14.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:14.015 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:15.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:14 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:15.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:14 smithi062 ceph-mon[64430]: osdmap e141: 8 total, 8 up, 8 in 2024-06-01T02:49:15.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:14 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:15.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:14 smithi062 ceph-mon[66884]: osdmap e141: 8 total, 8 up, 8 in 2024-06-01T02:49:15.014 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:49:14] "GET /metrics HTTP/1.1" 200 38376 "" "Prometheus/2.33.4" 2024-06-01T02:49:15.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:14 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:15.080 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:14 smithi146 ceph-mon[57708]: osdmap e141: 8 total, 8 up, 8 in 2024-06-01T02:49:15.760 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:49:16.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:15 smithi062 ceph-mon[64430]: pgmap v174: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 68/723 objects degraded (9.405%) 2024-06-01T02:49:16.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:15 smithi062 ceph-mon[66884]: pgmap v174: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 68/723 objects degraded (9.405%) 2024-06-01T02:49:16.042 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:15 smithi146 ceph-mon[57708]: pgmap v174: 161 pgs: 9 peering, 15 active+undersized+degraded, 29 active+undersized, 108 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 68/723 objects degraded (9.405%) 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (5m) 109s ago 10m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (5m) 4s ago 10m 51.5M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (5m) 109s ago 9m 54.1M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (4m) 4s ago 15m 453M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (5m) 109s ago 16m 507M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (4m) 109s ago 17m 50.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (4m) 4s ago 15m 38.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (4m) 109s ago 15m 37.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (5m) 109s ago 11m 18.2M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (5m) 4s ago 11m 17.4M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:49:16.447 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (3m) 109s ago 14m 53.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (2m) 109s ago 14m 44.4M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (2m) 109s ago 14m 42.0M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (113s) 109s ago 13m 11.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (78s) 4s ago 13m 52.4M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (44s) 4s ago 12m 50.9M 2992M 19.0.0-2946-g3d89accf b045ac545f82 31b032e2fbe8 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (9s) 4s ago 12m 14.7M 2992M 19.0.0-2946-g3d89accf b045ac545f82 5a8179547bde 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (12m) 4s ago 12m 63.8M 2992M 17.2.0 e1d6a67b021e ddc36ff5d28a 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (5m) 4s ago 11m 75.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (9m) 109s ago 9m 76.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (9m) 4s ago 9m 76.2M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (9m) 109s ago 9m 75.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:49:16.448 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (9m) 4s ago 9m 75.0M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:49:16.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:49:16.688+0000 7f15439f1700 -1 mgr.server reply reply (16) Device or resource busy unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-06-01T02:49:16.872 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:49:16.872 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:49:16.872 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:49:16.872 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 7 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-06-01T02:49:16.873 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 12 2024-06-01T02:49:16.874 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:49:16.874 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:49:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='client.44398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:17.219 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:49:17.219 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:49:17.219 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:49:17.219 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:49:17.219 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr" 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "12/23 daemons upgraded", 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:49:17.220 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:49:17.220 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:17.220 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='client.44398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:17.220 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.220 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='client.44392 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='client.44398 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:17.221 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:17.222 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:17.633 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN Degraded data redundancy: 41/723 objects degraded (5.671%), 8 pgs degraded 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 41/723 objects degraded (5.671%), 8 pgs degraded 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 4.e is active+undersized+degraded, acting [4,1] 2024-06-01T02:49:17.634 INFO:teuthology.orchestra.run.smithi062.stdout: pg 6.c is active+undersized+degraded, acting [3,5] 2024-06-01T02:49:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: pgmap v175: 161 pgs: 9 peering, 8 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41/723 objects degraded (5.671%) 2024-06-01T02:49:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-06-01T02:49:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/3635239109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:18.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: Health check update: Degraded data redundancy: 68/723 objects degraded (9.405%), 15 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: from='client.44416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/576881530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: pgmap v175: 161 pgs: 9 peering, 8 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41/723 objects degraded (5.671%) 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/3635239109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: Health check update: Degraded data redundancy: 68/723 objects degraded (9.405%), 15 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: from='client.44416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:18.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:17 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/576881530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:49:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: from='client.34476 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: pgmap v175: 161 pgs: 9 peering, 8 active+undersized+degraded, 18 active+undersized, 126 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 41/723 objects degraded (5.671%) 2024-06-01T02:49:18.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:18.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: Upgrade: unsafe to stop osd(s) at this time (13 PGs are or would become offline) 2024-06-01T02:49:18.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/3635239109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:18.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: Health check update: Degraded data redundancy: 68/723 objects degraded (9.405%), 15 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:18.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: from='client.44416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:18.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:17 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/576881530' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:49:19.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:18 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 8 pgs degraded) 2024-06-01T02:49:19.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:18 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:49:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:18 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 8 pgs degraded) 2024-06-01T02:49:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:18 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:49:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:18 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 41/723 objects degraded (5.671%), 8 pgs degraded) 2024-06-01T02:49:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:18 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:49:20.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:19 smithi146 ceph-mon[57708]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:49:20.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:19 smithi062 ceph-mon[64430]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:49:20.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:19 smithi062 ceph-mon[66884]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:49:22.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:21 smithi146 ceph-mon[57708]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-06-01T02:49:22.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:21 smithi062 ceph-mon[64430]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-06-01T02:49:22.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:21 smithi062 ceph-mon[66884]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-06-01T02:49:24.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:23 smithi062 ceph-mon[64430]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-06-01T02:49:24.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:23 smithi062 ceph-mon[66884]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-06-01T02:49:24.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:23 smithi146 ceph-mon[57708]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 233 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 0 B/s wr, 40 op/s 2024-06-01T02:49:25.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:24 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:25.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:24 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:25.014 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:49:24] "GET /metrics HTTP/1.1" 200 38376 "" "Prometheus/2.33.4" 2024-06-01T02:49:25.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:24 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:26.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:25 smithi146 ceph-mon[57708]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-06-01T02:49:26.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:25 smithi062 ceph-mon[64430]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-06-01T02:49:26.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:25 smithi062 ceph-mon[66884]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 49 op/s 2024-06-01T02:49:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:27 smithi146 ceph-mon[57708]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-06-01T02:49:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:28.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[64430]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-06-01T02:49:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:28.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:28.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[66884]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 0 B/s wr, 75 op/s 2024-06-01T02:49:28.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:28.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:28.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:30.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:29 smithi146 ceph-mon[57708]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-06-01T02:49:30.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:29 smithi062 ceph-mon[64430]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-06-01T02:49:30.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:29 smithi062 ceph-mon[66884]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-06-01T02:49:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:31 smithi146 ceph-mon[57708]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-06-01T02:49:32.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:31 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:31 smithi062 ceph-mon[64430]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-06-01T02:49:32.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:31 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:32.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:31 smithi062 ceph-mon[66884]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 0 B/s wr, 104 op/s 2024-06-01T02:49:32.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:31 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:32 smithi146 ceph-mon[57708]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:33.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:32 smithi146 ceph-mon[57708]: Upgrade: osd.7 is safe to restart 2024-06-01T02:49:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:32 smithi062 ceph-mon[64430]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:33.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:32 smithi062 ceph-mon[64430]: Upgrade: osd.7 is safe to restart 2024-06-01T02:49:33.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:32 smithi062 ceph-mon[66884]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-06-01T02:49:33.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:32 smithi062 ceph-mon[66884]: Upgrade: osd.7 is safe to restart 2024-06-01T02:49:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[66884]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-06-01T02:49:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:49:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:34.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[64430]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-06-01T02:49:34.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:34.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:49:34.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:33 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:34.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:33 smithi146 ceph-mon[57708]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 141 op/s 2024-06-01T02:49:34.202 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:34.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-06-01T02:49:34.203 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:33 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:34.869 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:49:34] "GET /metrics HTTP/1.1" 200 38527 "" "Prometheus/2.33.4" 2024-06-01T02:49:35.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:34 smithi146 ceph-mon[57708]: Upgrade: Updating osd.7 2024-06-01T02:49:35.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:34 smithi146 ceph-mon[57708]: Deploying daemon osd.7 on smithi146 2024-06-01T02:49:35.156 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:34 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-mon[64430]: Upgrade: Updating osd.7 2024-06-01T02:49:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-mon[64430]: Deploying daemon osd.7 on smithi146 2024-06-01T02:49:35.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:35.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-mon[66884]: Upgrade: Updating osd.7 2024-06-01T02:49:35.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-mon[66884]: Deploying daemon osd.7 on smithi146 2024-06-01T02:49:35.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:34 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:36.096 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:35 smithi146 ceph-mon[57708]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-06-01T02:49:36.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:35 smithi062 ceph-mon[64430]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-06-01T02:49:36.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:35 smithi062 ceph-mon[66884]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 0 B/s wr, 120 op/s 2024-06-01T02:49:36.706 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:36 smithi146 systemd[1]: Stopping Ceph osd.7 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:49:37.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:36 smithi146 ceph-mon[57708]: osd.7 marked itself down and dead 2024-06-01T02:49:37.206 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[38004]: 2024-06-01T02:49:36.726+0000 7fd01e6d5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:49:37.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[38004]: 2024-06-01T02:49:36.726+0000 7fd01e6d5700 -1 osd.7 141 *** Got signal Terminated *** 2024-06-01T02:49:37.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[38004]: 2024-06-01T02:49:36.726+0000 7fd01e6d5700 -1 osd.7 141 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:49:37.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:36 smithi062 ceph-mon[64430]: osd.7 marked itself down and dead 2024-06-01T02:49:37.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:36 smithi062 ceph-mon[66884]: osd.7 marked itself down and dead 2024-06-01T02:49:37.631 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:37 smithi146 podman[67304]: 2024-06-01 02:49:37.360612251 +0000 UTC m=+0.756849258 container died ddc36ff5d28aee54d976d62ea3986060e4c664ee3d8b26e070e4d273a4faa947 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.openshift.tags=base centos centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8) 2024-06-01T02:49:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:37 smithi146 ceph-mon[57708]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-06-01T02:49:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:37 smithi146 ceph-mon[57708]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:49:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:37 smithi146 ceph-mon[57708]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-06-01T02:49:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:37 smithi146 ceph-mon[57708]: osdmap e142: 8 total, 7 up, 8 in 2024-06-01T02:49:37.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:37 smithi146 podman[67304]: 2024-06-01 02:49:37.734931965 +0000 UTC m=+1.131168963 container cleanup ddc36ff5d28aee54d976d62ea3986060e4c664ee3d8b26e070e4d273a4faa947 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, release=754, version=8, GIT_CLEAN=True, name=centos-stream, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux ) 2024-06-01T02:49:37.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:37 smithi146 bash[67304]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7 2024-06-01T02:49:38.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[64430]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-06-01T02:49:38.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[64430]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:49:38.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[64430]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-06-01T02:49:38.165 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[64430]: osdmap e142: 8 total, 7 up, 8 in 2024-06-01T02:49:38.165 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[66884]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 100 KiB/s rd, 0 B/s wr, 166 op/s 2024-06-01T02:49:38.165 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[66884]: Health check failed: 1 osds down (OSD_DOWN) 2024-06-01T02:49:38.165 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[66884]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-06-01T02:49:38.165 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:37 smithi062 ceph-mon[66884]: osdmap e142: 8 total, 7 up, 8 in 2024-06-01T02:49:38.456 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:38 smithi146 podman[67331]: 2024-06-01 02:49:38.103177814 +0000 UTC m=+0.745367576 container remove ddc36ff5d28aee54d976d62ea3986060e4c664ee3d8b26e070e4d273a4faa947 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.buildah.version=1.19.8, distribution-scope=public, version=8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, ceph=True, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-06-01T02:49:38.765 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:38 smithi146 podman[67421]: 2024-06-01 02:49:38.414559865 +0000 UTC m=+0.017327251 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:49:38.765 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:38 smithi146 podman[67421]: 2024-06-01 02:49:38.512385584 +0000 UTC m=+0.115152954 container create ff775955afc35de82e30c41f4977c343e5656ab432c4984b2ee05e5624bf3642 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:49:39.071 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:38 smithi146 ceph-mon[57708]: osdmap e143: 8 total, 7 up, 8 in 2024-06-01T02:49:39.072 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:38 smithi146 podman[67421]: 2024-06-01 02:49:38.76552824 +0000 UTC m=+0.368295633 container init ff775955afc35de82e30c41f4977c343e5656ab432c4984b2ee05e5624bf3642 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-06-01T02:49:39.072 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:38 smithi146 podman[67421]: 2024-06-01 02:49:38.769302005 +0000 UTC m=+0.372069385 container start ff775955afc35de82e30c41f4977c343e5656ab432c4984b2ee05e5624bf3642 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:49:39.072 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:38 smithi146 podman[67421]: 2024-06-01 02:49:38.923459035 +0000 UTC m=+0.526226413 container attach ff775955afc35de82e30c41f4977c343e5656ab432c4984b2ee05e5624bf3642 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-06-01T02:49:39.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:38 smithi062 ceph-mon[64430]: osdmap e143: 8 total, 7 up, 8 in 2024-06-01T02:49:39.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:38 smithi062 ceph-mon[66884]: osdmap e143: 8 total, 7 up, 8 in 2024-06-01T02:49:39.367 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:39 smithi146 podman[67421]: 2024-06-01 02:49:39.072310582 +0000 UTC m=+0.675077965 container died ff775955afc35de82e30c41f4977c343e5656ab432c4984b2ee05e5624bf3642 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:49:39.622 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:39 smithi146 podman[67447]: 2024-06-01 02:49:39.601885964 +0000 UTC m=+0.520003872 container remove ff775955afc35de82e30c41f4977c343e5656ab432c4984b2ee05e5624bf3642 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0) 2024-06-01T02:49:39.895 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:39 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.7.service: Deactivated successfully. 2024-06-01T02:49:39.895 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:39 smithi146 systemd[1]: Stopped Ceph osd.7 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:49:39.896 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:39 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.7.service: Consumed 9.357s CPU time. 2024-06-01T02:49:39.896 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:39 smithi146 systemd[1]: Starting Ceph osd.7 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:49:40.182 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:39 smithi146 ceph-mon[57708]: pgmap v188: 161 pgs: 30 peering, 12 stale+active+clean, 119 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 198 op/s 2024-06-01T02:49:40.183 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:39 smithi146 ceph-mon[57708]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-01T02:49:40.183 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 podman[67545]: 2024-06-01 02:49:39.977633017 +0000 UTC m=+0.017494473 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:49:40.183 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 podman[67545]: 2024-06-01 02:49:40.077792379 +0000 UTC m=+0.117653831 container create 875f6aed50d7e707ed49627b6599e4eb2c91aaa1655ca92a6f2852a69e00f4b6 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-01T02:49:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:39 smithi062 ceph-mon[64430]: pgmap v188: 161 pgs: 30 peering, 12 stale+active+clean, 119 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 198 op/s 2024-06-01T02:49:40.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:39 smithi062 ceph-mon[64430]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-01T02:49:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:39 smithi062 ceph-mon[66884]: pgmap v188: 161 pgs: 30 peering, 12 stale+active+clean, 119 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 0 B/s wr, 198 op/s 2024-06-01T02:49:40.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:39 smithi062 ceph-mon[66884]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-06-01T02:49:40.456 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 podman[67545]: 2024-06-01 02:49:40.321815842 +0000 UTC m=+0.361677295 container init 875f6aed50d7e707ed49627b6599e4eb2c91aaa1655ca92a6f2852a69e00f4b6 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-06-01T02:49:40.456 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 podman[67545]: 2024-06-01 02:49:40.325334289 +0000 UTC m=+0.365195747 container start 875f6aed50d7e707ed49627b6599e4eb2c91aaa1655ca92a6f2852a69e00f4b6 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_CLEAN=True) 2024-06-01T02:49:40.456 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 podman[67545]: 2024-06-01 02:49:40.404960202 +0000 UTC m=+0.444821663 container attach 875f6aed50d7e707ed49627b6599e4eb2c91aaa1655ca92a6f2852a69e00f4b6 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-01T02:49:41.168 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 bash[67545]: --> ceph-volume raw activate successful for osd ID: 7 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate[67560]: --> ceph-volume raw activate successful for osd ID: 7 2024-06-01T02:49:41.169 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:40 smithi146 podman[67545]: 2024-06-01 02:49:40.801972723 +0000 UTC m=+0.841834182 container died 875f6aed50d7e707ed49627b6599e4eb2c91aaa1655ca92a6f2852a69e00f4b6 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-01T02:49:41.806 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:41 smithi146 podman[67692]: 2024-06-01 02:49:41.475676399 +0000 UTC m=+0.667520918 container remove 875f6aed50d7e707ed49627b6599e4eb2c91aaa1655ca92a6f2852a69e00f4b6 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-activate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-01T02:49:41.807 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:41 smithi146 podman[67751]: 2024-06-01 02:49:41.598082969 +0000 UTC m=+0.017407196 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:49:41.807 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:41 smithi146 podman[67751]: 2024-06-01 02:49:41.701351481 +0000 UTC m=+0.120675676 container create 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-01T02:49:42.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:41 smithi146 ceph-mon[57708]: pgmap v189: 161 pgs: 30 peering, 12 stale+active+clean, 119 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 142 op/s 2024-06-01T02:49:42.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:41 smithi146 podman[67751]: 2024-06-01 02:49:41.939187872 +0000 UTC m=+0.358512057 container init 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-06-01T02:49:42.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:41 smithi146 podman[67751]: 2024-06-01 02:49:41.942415732 +0000 UTC m=+0.361739935 container start 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=) 2024-06-01T02:49:42.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:41 smithi146 ceph-osd[67770]: -- 172.21.15.146:0/383528993 <== mon.2 v2:172.21.15.146:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ad8f8a7680 con 0x55ad8f870c00 2024-06-01T02:49:42.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:42 smithi146 bash[67751]: 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f 2024-06-01T02:49:42.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:42 smithi146 systemd[1]: Started Ceph osd.7 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:49:42.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:41 smithi062 ceph-mon[64430]: pgmap v189: 161 pgs: 30 peering, 12 stale+active+clean, 119 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 142 op/s 2024-06-01T02:49:42.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:41 smithi062 ceph-mon[66884]: pgmap v189: 161 pgs: 30 peering, 12 stale+active+clean, 119 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 85 KiB/s rd, 0 B/s wr, 142 op/s 2024-06-01T02:49:42.909 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:42 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:49:42.480+0000 7f2ca31587c0 -1 Falling back to public interface 2024-06-01T02:49:43.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:43.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:43.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:43.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:44.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:44 smithi146 ceph-mon[57708]: pgmap v190: 161 pgs: 26 active+undersized+degraded, 7 peering, 36 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:44.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:44 smithi146 ceph-mon[57708]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:44 smithi062 ceph-mon[64430]: pgmap v190: 161 pgs: 26 active+undersized+degraded, 7 peering, 36 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:44.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:44 smithi062 ceph-mon[64430]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:44 smithi062 ceph-mon[66884]: pgmap v190: 161 pgs: 26 active+undersized+degraded, 7 peering, 36 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 0 B/s wr, 128 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:44.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:44 smithi062 ceph-mon[66884]: Health check failed: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded (PG_DEGRADED) 2024-06-01T02:49:45.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:49:44] "GET /metrics HTTP/1.1" 200 38616 "" "Prometheus/2.33.4" 2024-06-01T02:49:45.349 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:45 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:45.349 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:45 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:49:45.197+0000 7f2ca31587c0 -1 osd.7 0 read_superblock omap replica is missing. 2024-06-01T02:49:45.349 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:45 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:49:45.228+0000 7f2ca31587c0 -1 osd.7 141 log_to_monitors true 2024-06-01T02:49:45.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:45 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:45 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:46.329 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:49:46 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:49:46.129+0000 7f2c945af700 -1 osd.7 141 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-01T02:49:46.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:46 smithi146 ceph-mon[57708]: pgmap v191: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 41 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:46.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:46 smithi146 ceph-mon[57708]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-06-01T02:49:46.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:46 smithi146 ceph-mon[57708]: from='osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:49:46.329 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:46 smithi146 ceph-mon[57708]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:49:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[64430]: pgmap v191: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 41 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[64430]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-06-01T02:49:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[64430]: from='osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:49:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[64430]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:49:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[66884]: pgmap v191: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 242 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 41 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[66884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-06-01T02:49:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[66884]: from='osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:49:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:46 smithi062 ceph-mon[66884]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-06-01T02:49:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:47 smithi146 ceph-mon[57708]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-01T02:49:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:47 smithi146 ceph-mon[57708]: osdmap e144: 8 total, 7 up, 8 in 2024-06-01T02:49:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:47 smithi146 ceph-mon[57708]: from='osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:47 smithi146 ceph-mon[57708]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:47 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:47.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:47 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[64430]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-01T02:49:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[64430]: osdmap e144: 8 total, 7 up, 8 in 2024-06-01T02:49:47.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[64430]: from='osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[64430]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[66884]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[66884]: osdmap e144: 8 total, 7 up, 8 in 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[66884]: from='osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[66884]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi146", "root=default"]}]: dispatch 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:47.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:47 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.010 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:49:48.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: pgmap v193: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:48.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:49:48.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157] boot 2024-06-01T02:49:48.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: osdmap e145: 8 total, 8 up, 8 in 2024-06-01T02:49:48.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:49:48.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[64430]: osdmap e146: 8 total, 8 up, 8 in 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: pgmap v193: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157] boot 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: osdmap e145: 8 total, 8 up, 8 in 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:48 smithi062 ceph-mon[66884]: osdmap e146: 8 total, 8 up, 8 in 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: pgmap v193: 161 pgs: 26 active+undersized+degraded, 43 active+undersized, 92 active+clean; 585 KiB data, 252 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: osd.7 [v2:172.21.15.146:6824/3374333157,v1:172.21.15.146:6825/3374333157] boot 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: osdmap e145: 8 total, 8 up, 8 in 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:48.389 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:48 smithi146 ceph-mon[57708]: osdmap e146: 8 total, 8 up, 8 in 2024-06-01T02:49:48.715 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:49:48.715 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (6m) 2m ago 11m 26.0M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (6m) 2s ago 10m 51.7M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (5m) 2m ago 10m 54.1M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (5m) 2s ago 15m 454M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (6m) 2m ago 17m 507M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (5m) 2m ago 17m 50.3M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (4m) 2s ago 15m 39.9M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (4m) 2m ago 16m 37.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (6m) 2m ago 11m 18.2M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (6m) 2s ago 11m 15.1M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (3m) 2m ago 15m 53.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (3m) 2m ago 14m 44.4M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (3m) 2m ago 14m 42.0M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (2m) 2m ago 14m 11.9M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:49:48.716 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (111s) 2s ago 13m 56.5M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (76s) 2s ago 13m 54.8M 2992M 19.0.0-2946-g3d89accf b045ac545f82 31b032e2fbe8 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (41s) 2s ago 13m 55.6M 2992M 19.0.0-2946-g3d89accf b045ac545f82 5a8179547bde 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (7s) 2s ago 12m 12.1M 2992M 19.0.0-2946-g3d89accf b045ac545f82 18402d725262 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (5m) 2s ago 11m 75.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (10m) 2m ago 10m 76.8M - 17.2.0 e1d6a67b021e 74f646c39079 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (10m) 2s ago 10m 76.5M - 17.2.0 e1d6a67b021e e1bbd54c735c 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (10m) 2m ago 10m 75.9M - 17.2.0 e1d6a67b021e e2da45b35eaf 2024-06-01T02:49:48.717 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (10m) 2s ago 10m 75.3M - 17.2.0 e1d6a67b021e e6a8843c5052 2024-06-01T02:49:49.125 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:49:49.125 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:49:49.125 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 8 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 13 2024-06-01T02:49:49.126 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:49:49.127 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:49:49.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:49 smithi146 ceph-mon[57708]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:49.393 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:49 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/256551516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:49.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:49 smithi062 ceph-mon[64430]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:49.477 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:49 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/256551516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:49.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:49 smithi062 ceph-mon[66884]: from='client.34506 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:49.477 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:49 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/256551516' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "osd" 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "13/23 daemons upgraded", 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading osd daemons", 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:49:49.478 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:49:49.891 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded 2024-06-01T02:49:49.891 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-06-01T02:49:49.891 INFO:teuthology.orchestra.run.smithi062.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-06-01T02:49:49.892 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-06-01T02:49:49.893 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-06-01T02:49:49.894 INFO:teuthology.orchestra.run.smithi062.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-06-01T02:49:49.894 INFO:teuthology.orchestra.run.smithi062.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-06-01T02:49:49.894 INFO:teuthology.orchestra.run.smithi062.stdout: pg 4.3 is active+undersized+degraded, acting [0,5] 2024-06-01T02:49:49.894 INFO:teuthology.orchestra.run.smithi062.stdout: pg 4.14 is active+undersized+degraded, acting [3,1] 2024-06-01T02:49:49.894 INFO:teuthology.orchestra.run.smithi062.stdout: pg 4.15 is active+undersized+degraded, acting [5,3] 2024-06-01T02:49:49.894 INFO:teuthology.orchestra.run.smithi062.stdout: pg 5.1 is active+undersized+degraded, acting [4,3] 2024-06-01T02:49:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[64430]: from='client.44440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[64430]: pgmap v196: 161 pgs: 4 peering, 26 active+undersized+degraded, 39 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[64430]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:50.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/589218162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:49:50.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[66884]: from='client.44440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:50.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[66884]: pgmap v196: 161 pgs: 4 peering, 26 active+undersized+degraded, 39 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:50.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[66884]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:50.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:50 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/589218162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:49:50.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:50 smithi146 ceph-mon[57708]: from='client.44440 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:50.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:50 smithi146 ceph-mon[57708]: pgmap v196: 161 pgs: 4 peering, 26 active+undersized+degraded, 39 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:50.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:50 smithi146 ceph-mon[57708]: from='client.34515 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:50.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:50 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/589218162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:49:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:51 smithi146 ceph-mon[57708]: from='client.44458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:49:51.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:51 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[64430]: from='client.44458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:51.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[66884]: from='client.44458 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:49:51.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:51 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: pgmap v197: 161 pgs: 4 peering, 26 active+undersized+degraded, 39 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all osd 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-06-01T02:49:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-06-01T02:49:52.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: Upgrade: Setting require_osd_release to 19 squid 2024-06-01T02:49:52.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:52 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-06-01T02:49:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: pgmap v197: 161 pgs: 4 peering, 26 active+undersized+degraded, 39 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:49:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all osd 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-06-01T02:49:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: Upgrade: Setting require_osd_release to 19 squid 2024-06-01T02:49:52.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: pgmap v197: 161 pgs: 4 peering, 26 active+undersized+degraded, 39 active+undersized, 92 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 95/723 objects degraded (13.140%) 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all osd 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-06-01T02:49:52.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: Upgrade: Setting require_osd_release to 19 squid 2024-06-01T02:49:52.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:52 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-06-01T02:49:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:53 smithi146 ceph-mon[57708]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-06-01T02:49:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-06-01T02:49:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:53 smithi146 ceph-mon[57708]: osdmap e147: 8 total, 8 up, 8 in 2024-06-01T02:49:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:53 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all mds 2024-06-01T02:49:53.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:53 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:53.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[66884]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[66884]: osdmap e147: 8 total, 8 up, 8 in 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all mds 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[64430]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[64430]: osdmap e147: 8 total, 8 up, 8 in 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all mds 2024-06-01T02:49:53.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:53 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:54.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:54 smithi146 ceph-mon[57708]: pgmap v199: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-01T02:49:54.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:54 smithi146 ceph-mon[57708]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded) 2024-06-01T02:49:54.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:54 smithi146 ceph-mon[57708]: Cluster is now healthy 2024-06-01T02:49:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:54 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:54 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:49:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:54 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:54.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[64430]: pgmap v199: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-01T02:49:54.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[64430]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded) 2024-06-01T02:49:54.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[64430]: Cluster is now healthy 2024-06-01T02:49:54.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:54.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:49:54.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:54.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[66884]: pgmap v199: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-06-01T02:49:54.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[66884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 95/723 objects degraded (13.140%), 26 pgs degraded) 2024-06-01T02:49:54.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[66884]: Cluster is now healthy 2024-06-01T02:49:54.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:54.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi062.lqlihd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:49:54.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:49:54.833 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:49:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:49:54] "GET /metrics HTTP/1.1" 200 38616 "" "Prometheus/2.33.4" 2024-06-01T02:49:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:55 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:55 smithi146 ceph-mon[57708]: Upgrade: Updating rgw.foo.smithi062.lqlihd (1/4) 2024-06-01T02:49:55.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:55 smithi146 ceph-mon[57708]: Deploying daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:49:55.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:55 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:55.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:55 smithi062 ceph-mon[64430]: Upgrade: Updating rgw.foo.smithi062.lqlihd (1/4) 2024-06-01T02:49:55.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:55 smithi062 ceph-mon[64430]: Deploying daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:49:55.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:55 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:49:55.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:55 smithi062 ceph-mon[66884]: Upgrade: Updating rgw.foo.smithi062.lqlihd (1/4) 2024-06-01T02:49:55.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:55 smithi062 ceph-mon[66884]: Deploying daemon rgw.foo.smithi062.lqlihd on smithi062 2024-06-01T02:49:56.298 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:56 smithi062 ceph-mon[64430]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-06-01T02:49:56.298 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:56 smithi062 ceph-mon[66884]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-06-01T02:49:56.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:56 smithi146 ceph-mon[57708]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 685 B/s rd, 0 op/s 2024-06-01T02:49:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:58 smithi062 ceph-mon[64430]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-06-01T02:49:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:58.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:58.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:58 smithi062 ceph-mon[66884]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-06-01T02:49:58.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:58.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:58.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:58 smithi146 ceph-mon[57708]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 604 B/s rd, 0 op/s 2024-06-01T02:49:58.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:58.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:49:59.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:59.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:59 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:59.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:49:59 smithi146 ceph-mon[57708]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:49:59.736 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:59.736 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:59 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:59.736 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:49:59 smithi062 ceph-mon[64430]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:49:59.736 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:59.736 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:59 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:49:59.736 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:49:59 smithi062 ceph-mon[66884]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:50:00.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:00 smithi146 ceph-mon[57708]: overall HEALTH_OK 2024-06-01T02:50:00.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:00 smithi146 ceph-mon[57708]: Upgrade: Updating rgw.smpl.smithi062.jsennv (2/4) 2024-06-01T02:50:00.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:00 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:00.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:00 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:00.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:00 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:00.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[66884]: overall HEALTH_OK 2024-06-01T02:50:00.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[66884]: Upgrade: Updating rgw.smpl.smithi062.jsennv (2/4) 2024-06-01T02:50:00.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:00.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:00.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:00.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[64430]: overall HEALTH_OK 2024-06-01T02:50:00.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[64430]: Upgrade: Updating rgw.smpl.smithi062.jsennv (2/4) 2024-06-01T02:50:00.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:00.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi062.jsennv", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:00.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:00 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:01 smithi146 ceph-mon[57708]: Deploying daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:50:01.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:01 smithi146 ceph-mon[57708]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:50:01.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:01 smithi062 ceph-mon[64430]: Deploying daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:50:01.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:01 smithi062 ceph-mon[64430]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:50:01.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:01 smithi062 ceph-mon[66884]: Deploying daemon rgw.smpl.smithi062.jsennv on smithi062 2024-06-01T02:50:01.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:01 smithi062 ceph-mon[66884]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 257 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:50:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:03 smithi146 ceph-mon[57708]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 99 B/s wr, 144 op/s 2024-06-01T02:50:03.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:03 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:04.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:03 smithi062 ceph-mon[66884]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 99 B/s wr, 144 op/s 2024-06-01T02:50:04.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:03 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:04.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:03 smithi062 ceph-mon[64430]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 99 B/s wr, 144 op/s 2024-06-01T02:50:04.014 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:03 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:04.682 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:50:04] "GET /metrics HTTP/1.1" 200 38612 "" "Prometheus/2.33.4" 2024-06-01T02:50:04.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:04 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:04.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:04 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:04.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:04 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:05.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:05.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:04 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:05.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:05 smithi146 ceph-mon[57708]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:06.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:05 smithi062 ceph-mon[64430]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:06.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:05 smithi062 ceph-mon[66884]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:07.306 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:07 smithi146 ceph-mon[57708]: Upgrade: Updating rgw.foo.smithi146.btivlw (3/4) 2024-06-01T02:50:07.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:07.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:07.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:07 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:07.307 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:07 smithi146 ceph-mon[57708]: Deploying daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:50:07.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[64430]: Upgrade: Updating rgw.foo.smithi146.btivlw (3/4) 2024-06-01T02:50:07.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:07.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:07.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:07.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[64430]: Deploying daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:50:07.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[66884]: Upgrade: Updating rgw.foo.smithi146.btivlw (3/4) 2024-06-01T02:50:07.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:07.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi146.btivlw", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:07.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:07.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:07 smithi062 ceph-mon[66884]: Deploying daemon rgw.foo.smithi146.btivlw on smithi146 2024-06-01T02:50:08.119 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:08 smithi146 ceph-mon[57708]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 170 B/s wr, 205 op/s 2024-06-01T02:50:08.415 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:08 smithi062 ceph-mon[64430]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 170 B/s wr, 205 op/s 2024-06-01T02:50:08.417 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:08 smithi062 ceph-mon[66884]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 265 MiB used, 715 GiB / 715 GiB avail; 130 KiB/s rd, 170 B/s wr, 205 op/s 2024-06-01T02:50:10.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:10 smithi146 ceph-mon[57708]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:10.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:10 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:10.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:10 smithi062 ceph-mon[64430]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:10.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:10 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:10.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:10 smithi062 ceph-mon[66884]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:10.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:10 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:11.315 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:11 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:11.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:11 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:11.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:11 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:12 smithi146 ceph-mon[57708]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:12.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:12.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[64430]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[66884]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 269 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi146.olokyu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-06-01T02:50:12.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:13.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:13 smithi146 ceph-mon[57708]: Upgrade: Updating rgw.smpl.smithi146.olokyu (4/4) 2024-06-01T02:50:13.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:13 smithi146 ceph-mon[57708]: Deploying daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:50:13.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:13 smithi062 ceph-mon[64430]: Upgrade: Updating rgw.smpl.smithi146.olokyu (4/4) 2024-06-01T02:50:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:13 smithi062 ceph-mon[64430]: Deploying daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:50:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:13 smithi062 ceph-mon[66884]: Upgrade: Updating rgw.smpl.smithi146.olokyu (4/4) 2024-06-01T02:50:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:13 smithi062 ceph-mon[66884]: Deploying daemon rgw.smpl.smithi146.olokyu on smithi146 2024-06-01T02:50:13.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:14.160 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:14 smithi146 ceph-mon[57708]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 235 KiB/s rd, 255 B/s wr, 371 op/s 2024-06-01T02:50:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:14 smithi062 ceph-mon[64430]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 235 KiB/s rd, 255 B/s wr, 371 op/s 2024-06-01T02:50:14.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:14 smithi062 ceph-mon[66884]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 235 KiB/s rd, 255 B/s wr, 371 op/s 2024-06-01T02:50:15.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:50:14] "GET /metrics HTTP/1.1" 200 38611 "" "Prometheus/2.33.4" 2024-06-01T02:50:15.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:15 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:15.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:15 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:15.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:15 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:16 smithi146 ceph-mon[57708]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:16.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:16 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:16.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:16 smithi062 ceph-mon[64430]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:16.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:16.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:16 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:16.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:16 smithi062 ceph-mon[66884]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 281 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:16.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:16.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:16 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:18 smithi146 ceph-mon[57708]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 170 B/s wr, 272 op/s 2024-06-01T02:50:18.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:18 smithi062 ceph-mon[64430]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 170 B/s wr, 272 op/s 2024-06-01T02:50:18.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:18 smithi062 ceph-mon[66884]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 285 MiB used, 715 GiB / 715 GiB avail; 173 KiB/s rd, 170 B/s wr, 272 op/s 2024-06-01T02:50:20.272 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:50:20.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:20 smithi146 ceph-mon[57708]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 85 B/s wr, 235 op/s 2024-06-01T02:50:20.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:20 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:20 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:20 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:20 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[64430]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 85 B/s wr, 235 op/s 2024-06-01T02:50:20.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[66884]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 149 KiB/s rd, 85 B/s wr, 235 op/s 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:20 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:20.996 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:50:20.996 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (6m) 1s ago 11m 24.8M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:50:20.996 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (6m) 1s ago 11m 52.0M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:50:20.996 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 running (6m) 1s ago 10m 59.6M - 3.5 e1d6a67b021e b1dc2ddc2e2e 2024-06-01T02:50:20.996 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (6m) 1s ago 16m 454M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:50:20.996 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (6m) 1s ago 18m 516M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (5m) 1s ago 18m 56.8M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (5m) 1s ago 16m 41.5M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (5m) 1s ago 16m 42.5M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (7m) 1s ago 12m 18.3M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (6m) 1s ago 12m 15.2M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (4m) 1s ago 15m 63.4M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (3m) 1s ago 15m 59.5M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (3m) 1s ago 15m 52.2M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (2m) 1s ago 14m 64.5M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (2m) 1s ago 14m 63.9M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (108s) 1s ago 13m 60.7M 2992M 19.0.0-2946-g3d89accf b045ac545f82 31b032e2fbe8 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (74s) 1s ago 13m 61.4M 2992M 19.0.0-2946-g3d89accf b045ac545f82 5a8179547bde 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (39s) 1s ago 13m 51.2M 2992M 19.0.0-2946-g3d89accf b045ac545f82 18402d725262 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (6m) 1s ago 12m 75.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (23s) 1s ago 11m 76.5M - 19.0.0-2946-g3d89accf b045ac545f82 0ea6344e3231 2024-06-01T02:50:20.997 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (11s) 1s ago 10m 76.3M - 19.0.0-2946-g3d89accf b045ac545f82 9a0b2bf0657f 2024-06-01T02:50:20.998 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (17s) 1s ago 10m 76.2M - 19.0.0-2946-g3d89accf b045ac545f82 ccb5d3763116 2024-06-01T02:50:20.998 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (5s) 1s ago 10m 78.1M - 19.0.0-2946-g3d89accf b045ac545f82 da2f6bd39e54 2024-06-01T02:50:21.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:21 smithi062 ceph-mon[64430]: from='client.44551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:21.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:21 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:21.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:21 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:21.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:21 smithi062 ceph-mon[66884]: from='client.44551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:21.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:21 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:21.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:21 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 8 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:21.423 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:50:21.424 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 4 2024-06-01T02:50:21.424 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:21.424 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:50:21.424 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 17 2024-06-01T02:50:21.424 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:50:21.424 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:50:21.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:21 smithi146 ceph-mon[57708]: from='client.44551 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:21.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:21 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:21.447 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:21 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:21.789 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:50:21.789 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:50:21.789 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw" 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "17/23 daemons upgraded", 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Currently upgrading rgw daemons", 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": false 2024-06-01T02:50:21.790 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:50:22.218 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_OK 2024-06-01T02:50:22.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:22 smithi146 ceph-mon[57708]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 85 B/s wr, 192 op/s 2024-06-01T02:50:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:22 smithi146 ceph-mon[57708]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:22 smithi146 ceph-mon[57708]: from='client.44563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:22 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:22 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:22.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:22 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/4185425355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[64430]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 85 B/s wr, 192 op/s 2024-06-01T02:50:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[64430]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[64430]: from='client.44563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/4185425355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[66884]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 122 KiB/s rd, 85 B/s wr, 192 op/s 2024-06-01T02:50:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[66884]: from='client.34629 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[66884]: from='client.44563 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:22 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/4185425355' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:23.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:23 smithi146 ceph-mon[57708]: from='client.44575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:23.384 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:23 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1040127518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:50:23.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:23 smithi062 ceph-mon[64430]: from='client.44575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:23.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:23 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1040127518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:50:23.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:23 smithi062 ceph-mon[66884]: from='client.44575 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:23.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:23 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1040127518' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:50:24.381 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-mon[64430]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:24.381 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:24.381 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:24.382 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-mon[66884]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:24.382 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:24.382 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:24 smithi146 ceph-mon[57708]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 157 KiB/s rd, 170 B/s wr, 248 op/s 2024-06-01T02:50:24.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:24 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:24.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:24 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:24.706 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:50:24] "GET /metrics HTTP/1.1" 200 38611 "" "Prometheus/2.33.4" 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: Detected new or changed devices on smithi146 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:25.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi062.lqlihd"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi062.lqlihd"}]': finished 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi146.btivlw"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi146.btivlw"}]': finished 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi062.jsennv"}]: dispatch 2024-06-01T02:50:25.458 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi062.jsennv"}]': finished 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi146.olokyu"}]: dispatch 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi146.olokyu"}]': finished 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.459 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:25 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: Detected new or changed devices on smithi146 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi062.lqlihd"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi062.lqlihd"}]': finished 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi146.btivlw"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi146.btivlw"}]': finished 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi062.jsennv"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi062.jsennv"}]': finished 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi146.olokyu"}]: dispatch 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi146.olokyu"}]': finished 2024-06-01T02:50:25.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: Detected new or changed devices on smithi146 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.516 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi062.lqlihd"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi062.lqlihd"}]': finished 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi146.btivlw"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.foo.smithi146.btivlw"}]': finished 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi062.jsennv"}]: dispatch 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi062.jsennv"}]': finished 2024-06-01T02:50:25.517 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi146.olokyu"}]: dispatch 2024-06-01T02:50:25.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw.smpl.smithi146.olokyu"}]': finished 2024-06-01T02:50:25.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:25.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:25.518 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:25 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:26.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:26 smithi146 ceph-mon[57708]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:26.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:26 smithi146 ceph-mon[57708]: Detected new or changed devices on smithi062 2024-06-01T02:50:26.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:26 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all rgw 2024-06-01T02:50:26.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:26 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all rbd-mirror 2024-06-01T02:50:26.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:26 smithi146 ceph-mon[57708]: Upgrade: Setting container_image for all ceph-exporter 2024-06-01T02:50:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[64430]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[64430]: Detected new or changed devices on smithi062 2024-06-01T02:50:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all rgw 2024-06-01T02:50:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all rbd-mirror 2024-06-01T02:50:26.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[64430]: Upgrade: Setting container_image for all ceph-exporter 2024-06-01T02:50:26.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[66884]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:26.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[66884]: Detected new or changed devices on smithi062 2024-06-01T02:50:26.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all rgw 2024-06-01T02:50:26.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all rbd-mirror 2024-06-01T02:50:26.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:26 smithi062 ceph-mon[66884]: Upgrade: Setting container_image for all ceph-exporter 2024-06-01T02:50:27.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: Upgrade: Updating iscsi.foo.smithi062.ywlcmk 2024-06-01T02:50:27.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:27.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: Deploying daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: Upgrade: Updating iscsi.foo.smithi062.ywlcmk 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: Deploying daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:27.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:27 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: Upgrade: Updating iscsi.foo.smithi062.ywlcmk 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi062.ywlcmk", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: Deploying daemon iscsi.foo.smithi062.ywlcmk on smithi062 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 79 KiB/s rd, 85 B/s wr, 124 op/s 2024-06-01T02:50:27.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:27 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:29.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:29 smithi146 ceph-mon[57708]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 100 op/s 2024-06-01T02:50:30.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:29 smithi062 ceph-mon[66884]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 100 op/s 2024-06-01T02:50:30.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:29 smithi062 ceph-mon[64430]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 85 B/s wr, 100 op/s 2024-06-01T02:50:31.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:31 smithi146 ceph-mon[57708]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 85 B/s wr, 56 op/s 2024-06-01T02:50:32.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:31 smithi062 ceph-mon[64430]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 85 B/s wr, 56 op/s 2024-06-01T02:50:32.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:31 smithi062 ceph-mon[66884]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 85 B/s wr, 56 op/s 2024-06-01T02:50:33.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:33 smithi146 ceph-mon[57708]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 85 B/s wr, 57 op/s 2024-06-01T02:50:34.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:33 smithi062 ceph-mon[64430]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 85 B/s wr, 57 op/s 2024-06-01T02:50:34.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:33 smithi062 ceph-mon[66884]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 85 B/s wr, 57 op/s 2024-06-01T02:50:34.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:34 smithi146 ceph-mon[57708]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:35.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:34 smithi062 ceph-mon[64430]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:35.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:50:34] "GET /metrics HTTP/1.1" 200 38673 "" "Prometheus/2.33.4" 2024-06-01T02:50:35.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:34 smithi062 ceph-mon[66884]: from='client.25228 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:50:35.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:35 smithi146 ceph-mon[57708]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:50:36.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:35 smithi062 ceph-mon[64430]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:50:36.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:35 smithi062 ceph-mon[66884]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:50:37.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:37 smithi146 ceph-mon[57708]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:50:38.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:37 smithi062 ceph-mon[64430]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:50:38.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:37 smithi062 ceph-mon[66884]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:50:39.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:39 smithi146 ceph-mon[57708]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:50:40.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:39 smithi062 ceph-mon[64430]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:50:40.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:39 smithi062 ceph-mon[66884]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:50:40.741 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:50:40.381+0000 7f150cd74700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. 2024-06-01T02:50:41.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:41 smithi146 ceph-mon[57708]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. 2024-06-01T02:50:41.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:41 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:41.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:41 smithi146 ceph-mon[57708]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-06-01T02:50:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:41 smithi062 ceph-mon[64430]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. 2024-06-01T02:50:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:41 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:41 smithi062 ceph-mon[64430]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-06-01T02:50:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:41 smithi062 ceph-mon[66884]: Upgrade: Paused due to UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. 2024-06-01T02:50:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:41 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:41 smithi062 ceph-mon[66884]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2024-06-01T02:50:42.394 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:42 smithi146 ceph-mon[57708]: Health check failed: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-06-01T02:50:42.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:42 smithi062 ceph-mon[64430]: Health check failed: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-06-01T02:50:42.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:42 smithi062 ceph-mon[66884]: Health check failed: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. (UPGRADE_REDEPLOY_DAEMON) 2024-06-01T02:50:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:43 smithi146 ceph-mon[57708]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-06-01T02:50:43.956 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:43.957 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:43 smithi062 ceph-mon[64430]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-06-01T02:50:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:44.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:44.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:43 smithi062 ceph-mon[66884]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 954 B/s rd, 0 op/s 2024-06-01T02:50:44.013 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:44.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:45.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:50:44] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-06-01T02:50:45.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:45.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:45.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:45.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:45.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:50:45.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:50:45.515 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:46.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:46 smithi146 ceph-mon[57708]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 521 B/s rd, 0 op/s 2024-06-01T02:50:46.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:46 smithi146 ceph-mon[57708]: Checking dashboard <-> RGW credentials 2024-06-01T02:50:46.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:46 smithi146 ceph-mon[57708]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-06-01T02:50:46.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:46 smithi146 ceph-mon[57708]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-06-01T02:50:46.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:46 smithi146 ceph-mon[57708]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:50:46.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[64430]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 521 B/s rd, 0 op/s 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[64430]: Checking dashboard <-> RGW credentials 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[64430]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[64430]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[64430]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[66884]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 521 B/s rd, 0 op/s 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[66884]: Checking dashboard <-> RGW credentials 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[66884]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 605 B/s rd, 0 op/s 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[66884]: pgmap v227: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 754 B/s rd, 0 op/s 2024-06-01T02:50:46.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:46 smithi062 ceph-mon[66884]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:50:48.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:48 smithi146 ceph-mon[57708]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-06-01T02:50:48.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:48 smithi062 ceph-mon[64430]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-06-01T02:50:48.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:48 smithi062 ceph-mon[66884]: pgmap v228: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2 op/s 2024-06-01T02:50:50.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:50 smithi146 ceph-mon[57708]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-06-01T02:50:50.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:50 smithi062 ceph-mon[64430]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-06-01T02:50:50.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:50 smithi062 ceph-mon[66884]: pgmap v229: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.8 KiB/s rd, 2 op/s 2024-06-01T02:50:52.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:52 smithi146 ceph-mon[57708]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-06-01T02:50:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:52 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1336846385' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:50:52.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:52 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/2954762133' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1947009494"}]: dispatch 2024-06-01T02:50:52.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:52 smithi062 ceph-mon[64430]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-06-01T02:50:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:52 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1336846385' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:50:52.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:52 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/2954762133' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1947009494"}]: dispatch 2024-06-01T02:50:52.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:52 smithi062 ceph-mon[66884]: pgmap v230: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 2 op/s 2024-06-01T02:50:52.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:52 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1336846385' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-06-01T02:50:52.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:52 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/2954762133' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1947009494"}]: dispatch 2024-06-01T02:50:52.607 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:50:52.961 INFO:teuthology.orchestra.run.smithi062.stdout:"Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed." 2024-06-01T02:50:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:53 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/2954762133' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1947009494"}]': finished 2024-06-01T02:50:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:53 smithi062 ceph-mon[66884]: osdmap e148: 8 total, 8 up, 8 in 2024-06-01T02:50:53.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:53 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/256294219' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3686179755"}]: dispatch 2024-06-01T02:50:53.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:53 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/2954762133' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1947009494"}]': finished 2024-06-01T02:50:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:53 smithi062 ceph-mon[64430]: osdmap e148: 8 total, 8 up, 8 in 2024-06-01T02:50:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:53 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/256294219' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3686179755"}]: dispatch 2024-06-01T02:50:53.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:53 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/2954762133' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6800/1947009494"}]': finished 2024-06-01T02:50:53.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:53 smithi146 ceph-mon[57708]: osdmap e148: 8 total, 8 up, 8 in 2024-06-01T02:50:53.427 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:53 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/256294219' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3686179755"}]: dispatch 2024-06-01T02:50:53.820 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch ps' 2024-06-01T02:50:54.433 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: from='client.44659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:54.433 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-06-01T02:50:54.433 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: from='client.44665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:54.433 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/256294219' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3686179755"}]': finished 2024-06-01T02:50:54.433 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: osdmap e149: 8 total, 8 up, 8 in 2024-06-01T02:50:54.433 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/2177311179' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]: dispatch 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[64430]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]: dispatch 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: from='client.44659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: from='client.44665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/256294219' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3686179755"}]': finished 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: osdmap e149: 8 total, 8 up, 8 in 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/2177311179' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]: dispatch 2024-06-01T02:50:54.434 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-mon[66884]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]: dispatch 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: from='client.44659 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: pgmap v232: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 2.9 KiB/s rd, 3 op/s 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: from='client.44665 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/256294219' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3686179755"}]': finished 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: osdmap e149: 8 total, 8 up, 8 in 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/2177311179' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]: dispatch 2024-06-01T02:50:54.457 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:54 smithi146 ceph-mon[57708]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]: dispatch 2024-06-01T02:50:54.763 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:50:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:50:54] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-06-01T02:50:55.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:55 smithi062 ceph-mon[64430]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]': finished 2024-06-01T02:50:55.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:55 smithi062 ceph-mon[64430]: osdmap e150: 8 total, 8 up, 8 in 2024-06-01T02:50:55.446 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:55 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/51955683' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2451113228"}]: dispatch 2024-06-01T02:50:55.446 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:55 smithi062 ceph-mon[66884]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]': finished 2024-06-01T02:50:55.446 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:55 smithi062 ceph-mon[66884]: osdmap e150: 8 total, 8 up, 8 in 2024-06-01T02:50:55.446 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:55 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/51955683' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2451113228"}]: dispatch 2024-06-01T02:50:55.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:55 smithi146 ceph-mon[57708]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/4196536216"}]': finished 2024-06-01T02:50:55.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:55 smithi146 ceph-mon[57708]: osdmap e150: 8 total, 8 up, 8 in 2024-06-01T02:50:55.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:55 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/51955683' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2451113228"}]: dispatch 2024-06-01T02:50:56.000 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:50:56.000 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (7m) 11s ago 12m 24.8M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (7m) 11s ago 12m 54.2M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 unknown 11s ago 11m - - 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (6m) 11s ago 16m 454M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (7m) 11s ago 18m 517M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (6m) 11s ago 18m 57.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (5m) 11s ago 17m 42.5M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (5m) 11s ago 17m 43.9M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (7m) 11s ago 12m 18.5M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (7m) 11s ago 12m 15.7M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (4m) 11s ago 16m 63.6M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (4m) 11s ago 16m 58.6M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (4m) 11s ago 15m 51.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:50:56.001 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (3m) 11s ago 15m 64.7M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (2m) 11s ago 14m 64.1M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (2m) 11s ago 14m 59.9M 2992M 19.0.0-2946-g3d89accf b045ac545f82 31b032e2fbe8 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (109s) 11s ago 14m 61.6M 2992M 19.0.0-2946-g3d89accf b045ac545f82 5a8179547bde 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (74s) 11s ago 13m 50.5M 2992M 19.0.0-2946-g3d89accf b045ac545f82 18402d725262 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (6m) 11s ago 13m 75.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (58s) 11s ago 11m 76.8M - 19.0.0-2946-g3d89accf b045ac545f82 0ea6344e3231 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (46s) 11s ago 11m 76.7M - 19.0.0-2946-g3d89accf b045ac545f82 9a0b2bf0657f 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (52s) 11s ago 11m 76.5M - 19.0.0-2946-g3d89accf b045ac545f82 ccb5d3763116 2024-06-01T02:50:56.002 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (40s) 11s ago 11m 78.4M - 19.0.0-2946-g3d89accf b045ac545f82 da2f6bd39e54 2024-06-01T02:50:56.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[64430]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-06-01T02:50:56.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/51955683' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2451113228"}]': finished 2024-06-01T02:50:56.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[64430]: osdmap e151: 8 total, 8 up, 8 in 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1791884293' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]: dispatch 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[64430]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]: dispatch 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[66884]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/51955683' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2451113228"}]': finished 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[66884]: osdmap e151: 8 total, 8 up, 8 in 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1791884293' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]: dispatch 2024-06-01T02:50:56.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:56 smithi062 ceph-mon[66884]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]: dispatch 2024-06-01T02:50:56.685 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph versions' 2024-06-01T02:50:56.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:56 smithi146 ceph-mon[57708]: pgmap v235: 161 pgs: 161 active+clean; 585 KiB data, 289 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-06-01T02:50:56.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:56 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/51955683' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/2451113228"}]': finished 2024-06-01T02:50:56.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:56 smithi146 ceph-mon[57708]: osdmap e151: 8 total, 8 up, 8 in 2024-06-01T02:50:56.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:56 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1791884293' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]: dispatch 2024-06-01T02:50:56.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:56 smithi146 ceph-mon[57708]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]: dispatch 2024-06-01T02:50:57.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[66884]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:57.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[66884]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]': finished 2024-06-01T02:50:57.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[66884]: osdmap e152: 8 total, 8 up, 8 in 2024-06-01T02:50:57.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/403609855' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3488130991"}]: dispatch 2024-06-01T02:50:57.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[64430]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:57.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[64430]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]': finished 2024-06-01T02:50:57.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[64430]: osdmap e152: 8 total, 8 up, 8 in 2024-06-01T02:50:57.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:57 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/403609855' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3488130991"}]: dispatch 2024-06-01T02:50:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:57 smithi146 ceph-mon[57708]: from='client.34758 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:50:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:57 smithi146 ceph-mon[57708]: from='client.? ' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:6801/1947009494"}]': finished 2024-06-01T02:50:57.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:57 smithi146 ceph-mon[57708]: osdmap e152: 8 total, 8 up, 8 in 2024-06-01T02:50:57.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:57 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/403609855' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3488130991"}]: dispatch 2024-06-01T02:50:58.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[64430]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/403609855' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3488130991"}]': finished 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[64430]: osdmap e153: 8 total, 8 up, 8 in 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[66884]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/403609855' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3488130991"}]': finished 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[66884]: osdmap e153: 8 total, 8 up, 8 in 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:58.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:58 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:58.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:58 smithi146 ceph-mon[57708]: pgmap v238: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail 2024-06-01T02:50:58.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:58 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/403609855' entity='client.iscsi.foo.smithi062.ywlcmk' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.62:0/3488130991"}]': finished 2024-06-01T02:50:58.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:58 smithi146 ceph-mon[57708]: osdmap e153: 8 total, 8 up, 8 in 2024-06-01T02:50:58.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:50:58.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:58 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:50:58.724 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:50:58.724 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:50:58.724 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:50:58.724 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:58.724 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 8 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 4 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 17 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:50:58.725 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:50:59.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:50:59 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1617038141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:59.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:50:59 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1617038141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:50:59.591 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-06-01T02:50:59.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:50:59 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1617038141' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:51:00.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:00 smithi062 ceph-mon[64430]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:00.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:00 smithi062 ceph-mon[66884]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:00.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:00 smithi146 ceph-mon[57708]: pgmap v240: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:01.194 INFO:teuthology.orchestra.run.smithi062.stdout:wait for servicemap items w/ changing names to refresh 2024-06-01T02:51:01.929 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'sleep 60' 2024-06-01T02:51:02.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:02 smithi062 ceph-mon[64430]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:02.480 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:02 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:02.481 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:02 smithi062 ceph-mon[66884]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:02.481 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:02 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:02.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:02 smithi146 ceph-mon[57708]: pgmap v241: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:02.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:02 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:04.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:04 smithi062 ceph-mon[66884]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:51:04.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:04 smithi062 ceph-mon[64430]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:51:04.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:04 smithi146 ceph-mon[57708]: pgmap v242: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-06-01T02:51:05.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:51:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:51:04] "GET /metrics HTTP/1.1" 200 38755 "" "Prometheus/2.33.4" 2024-06-01T02:51:06.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:06 smithi062 ceph-mon[64430]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:06.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:06 smithi062 ceph-mon[66884]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:06.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:06 smithi146 ceph-mon[57708]: pgmap v243: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:08.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:08 smithi062 ceph-mon[64430]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:51:08.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:08 smithi062 ceph-mon[66884]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:51:08.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:08 smithi146 ceph-mon[57708]: pgmap v244: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:51:10.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:10 smithi062 ceph-mon[64430]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-06-01T02:51:10.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:10 smithi062 ceph-mon[66884]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-06-01T02:51:10.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:10 smithi146 ceph-mon[57708]: pgmap v245: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 888 B/s rd, 0 op/s 2024-06-01T02:51:12.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:12 smithi062 ceph-mon[64430]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:12.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:12 smithi062 ceph-mon[66884]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:12.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:12 smithi146 ceph-mon[57708]: pgmap v246: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:13 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:13.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:13 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:13 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:13.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:13 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:13.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:13 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:13.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:13 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:14.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:14 smithi062 ceph-mon[64430]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:14.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:14 smithi062 ceph-mon[66884]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:14.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:14 smithi146 ceph-mon[57708]: pgmap v247: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:15.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:51:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:51:14] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-06-01T02:51:16.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:16 smithi062 ceph-mon[64430]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:16.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:16 smithi062 ceph-mon[66884]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:16.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:16 smithi146 ceph-mon[57708]: pgmap v248: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:18.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:18 smithi062 ceph-mon[64430]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:18.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:18 smithi062 ceph-mon[66884]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:18.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:18 smithi146 ceph-mon[57708]: pgmap v249: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:20.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:20 smithi062 ceph-mon[64430]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:20.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:20 smithi062 ceph-mon[66884]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:20.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:20 smithi146 ceph-mon[57708]: pgmap v250: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:22.509 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:22 smithi146 ceph-mon[57708]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:22.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:22 smithi062 ceph-mon[64430]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:22.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:22 smithi062 ceph-mon[66884]: pgmap v251: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:23.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:23 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:23.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:23 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:23.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:23 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:24.592 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:24 smithi062 ceph-mon[64430]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:24.593 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:24 smithi062 ceph-mon[66884]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:24.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:24 smithi146 ceph-mon[57708]: pgmap v252: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:25.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:51:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:51:24] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-06-01T02:51:26.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:26 smithi146 ceph-mon[57708]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:26.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:26 smithi062 ceph-mon[64430]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:26.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:26 smithi062 ceph-mon[66884]: pgmap v253: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:28 smithi146 ceph-mon[57708]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:28.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:28.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:28 smithi062 ceph-mon[64430]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:28.764 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:28.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:28 smithi062 ceph-mon[66884]: pgmap v254: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:28.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:30.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:30 smithi146 ceph-mon[57708]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:30.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:30 smithi062 ceph-mon[64430]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:30.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:30 smithi062 ceph-mon[66884]: pgmap v255: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:32.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:32 smithi146 ceph-mon[57708]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:32.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:32 smithi062 ceph-mon[64430]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:32.763 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:32 smithi062 ceph-mon[66884]: pgmap v256: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:33.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:33 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:33.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:33 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:33.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:33 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:34.592 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:34 smithi062 ceph-mon[64430]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:34.592 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:34 smithi062 ceph-mon[66884]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:34 smithi146 ceph-mon[57708]: pgmap v257: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:35.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:51:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:51:34] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-06-01T02:51:36.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:36 smithi146 ceph-mon[57708]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:36.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:36 smithi062 ceph-mon[64430]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:36.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:36 smithi062 ceph-mon[66884]: pgmap v258: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:38.566 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:38 smithi062 ceph-mon[64430]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:38.566 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:38 smithi062 ceph-mon[66884]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:38.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:38 smithi146 ceph-mon[57708]: pgmap v259: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:40.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:40 smithi146 ceph-mon[57708]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:40.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:40 smithi062 ceph-mon[64430]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:40.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:40 smithi062 ceph-mon[66884]: pgmap v260: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:41.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:41 smithi146 ceph-mon[57708]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:41.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:41 smithi062 ceph-mon[64430]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:41.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:41 smithi062 ceph-mon[66884]: pgmap v261: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:51:42.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:42 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:42.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:42 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:42.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:42 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:43.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:43 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:43.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:43 smithi146 ceph-mon[57708]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:43.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:43 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:43.763 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:43 smithi062 ceph-mon[64430]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:43.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:43 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:43.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:43 smithi062 ceph-mon[66884]: pgmap v262: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:45.013 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:51:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:51:44] "GET /metrics HTTP/1.1" 200 38757 "" "Prometheus/2.33.4" 2024-06-01T02:51:46.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:45 smithi146 ceph-mon[57708]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-01T02:51:46.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:51:46.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:51:46.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:45 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:51:46.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[64430]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-01T02:51:46.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:51:46.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:51:46.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:51:46.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[66884]: pgmap v263: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-06-01T02:51:46.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-01T02:51:46.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-01T02:51:46.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:45 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' 2024-06-01T02:51:47.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:46 smithi146 ceph-mon[57708]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-06-01T02:51:47.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:46 smithi146 ceph-mon[57708]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:47.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:46 smithi062 ceph-mon[66884]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-06-01T02:51:47.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:46 smithi062 ceph-mon[66884]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:47.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:46 smithi062 ceph-mon[64430]: pgmap v264: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 953 B/s rd, 0 op/s 2024-06-01T02:51:47.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:46 smithi062 ceph-mon[64430]: pgmap v265: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:49.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:48 smithi146 ceph-mon[57708]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:49.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:48 smithi062 ceph-mon[64430]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:49.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:48 smithi062 ceph-mon[66884]: pgmap v266: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:51.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:50 smithi146 ceph-mon[57708]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:51.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:50 smithi062 ceph-mon[64430]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:51.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:50 smithi062 ceph-mon[66884]: pgmap v267: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:53.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:52 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:53.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:52 smithi146 ceph-mon[57708]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-06-01T02:51:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:52 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:53.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:52 smithi062 ceph-mon[64430]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-06-01T02:51:53.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:52 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:51:53.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:52 smithi062 ceph-mon[66884]: pgmap v268: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-06-01T02:51:55.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:54 smithi062 ceph-mon[64430]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:55.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:54 smithi062 ceph-mon[66884]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:55.014 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:51:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:51:54] "GET /metrics HTTP/1.1" 200 38757 "" "Prometheus/2.33.4" 2024-06-01T02:51:55.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:54 smithi146 ceph-mon[57708]: pgmap v269: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-06-01T02:51:57.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:56 smithi146 ceph-mon[57708]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:51:57.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:56 smithi062 ceph-mon[64430]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:51:57.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:56 smithi062 ceph-mon[66884]: pgmap v270: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-06-01T02:51:58.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:57 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:58.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:57 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:58.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:57 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:51:59.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:51:58 smithi146 ceph-mon[57708]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:59.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:51:58 smithi062 ceph-mon[64430]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:51:59.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:51:58 smithi062 ceph-mon[66884]: pgmap v271: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:01.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:00 smithi146 ceph-mon[57708]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:01.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:00 smithi062 ceph-mon[64430]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:01.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:00 smithi062 ceph-mon[66884]: pgmap v272: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:03.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:02 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:03.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:02 smithi146 ceph-mon[57708]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:03.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:02 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:03.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:02 smithi062 ceph-mon[66884]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:03.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:02 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:03.264 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:02 smithi062 ceph-mon[64430]: pgmap v273: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:04.492 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch ps' 2024-06-01T02:52:04.764 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:52:04] "GET /metrics HTTP/1.1" 200 38756 "" "Prometheus/2.33.4" 2024-06-01T02:52:05.164 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:04 smithi062 ceph-mon[64430]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:05.164 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:04 smithi062 ceph-mon[66884]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:05.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:04 smithi146 ceph-mon[57708]: pgmap v274: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:06.575 INFO:teuthology.orchestra.run.smithi062.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-06-01T02:52:06.575 INFO:teuthology.orchestra.run.smithi062.stdout:alertmanager.a smithi062 *:9093,9094 running (8m) 82s ago 13m 24.8M - 0.23.0 ba2b418f427c f2f8ba65c523 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:grafana.a smithi146 *:3000 running (8m) 82s ago 13m 54.2M - 8.3.5 dad864ee21e9 138c911bd936 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:iscsi.foo.smithi062.ywlcmk smithi062 unknown 82s ago 12m - - 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.x smithi146 *:8443,9283,8765 running (7m) 82s ago 18m 454M - 19.0.0-2946-g3d89accf b045ac545f82 e90d7c11bbbf 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:mgr.y smithi062 *:8443,9283,8765 running (8m) 82s ago 19m 517M - 19.0.0-2946-g3d89accf b045ac545f82 65df26cb40cc 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:mon.a smithi062 running (7m) 82s ago 19m 57.2M 2048M 19.0.0-2946-g3d89accf b045ac545f82 f8034e39fc3a 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:mon.b smithi146 running (6m) 82s ago 18m 42.5M 2048M 19.0.0-2946-g3d89accf b045ac545f82 b8af55737ea2 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:mon.c smithi062 running (7m) 82s ago 18m 43.9M 2048M 19.0.0-2946-g3d89accf b045ac545f82 7ee52beadd88 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.a smithi062 *:9100 running (8m) 82s ago 14m 18.5M - 1.3.1 1dbe0e931976 eb8c079db150 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:node-exporter.b smithi146 *:9100 running (8m) 82s ago 14m 15.7M - 1.3.1 1dbe0e931976 c725f8ac02c4 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:osd.0 smithi062 running (5m) 82s ago 17m 63.6M 2736M 19.0.0-2946-g3d89accf b045ac545f82 fa2b42ebab36 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:osd.1 smithi062 running (5m) 82s ago 17m 58.6M 2736M 19.0.0-2946-g3d89accf b045ac545f82 0225f93df933 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:osd.2 smithi062 running (5m) 82s ago 16m 51.8M 2736M 19.0.0-2946-g3d89accf b045ac545f82 c4ad797c1b5a 2024-06-01T02:52:06.576 INFO:teuthology.orchestra.run.smithi062.stdout:osd.3 smithi062 running (4m) 82s ago 16m 64.7M 2736M 19.0.0-2946-g3d89accf b045ac545f82 2979765bc60d 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.4 smithi146 running (4m) 82s ago 16m 64.1M 2992M 19.0.0-2946-g3d89accf b045ac545f82 b204dde364d5 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.5 smithi146 running (3m) 82s ago 15m 59.9M 2992M 19.0.0-2946-g3d89accf b045ac545f82 31b032e2fbe8 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.6 smithi146 running (2m) 82s ago 15m 61.6M 2992M 19.0.0-2946-g3d89accf b045ac545f82 5a8179547bde 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:osd.7 smithi146 running (2m) 82s ago 14m 50.5M 2992M 19.0.0-2946-g3d89accf b045ac545f82 18402d725262 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:prometheus.a smithi146 *:9095 running (7m) 82s ago 14m 75.2M - 2.33.4 514e6a882f6e 2565089ff6fe 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi062.lqlihd smithi062 *:8000 running (2m) 82s ago 12m 76.8M - 19.0.0-2946-g3d89accf b045ac545f82 0ea6344e3231 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.foo.smithi146.btivlw smithi146 *:8000 running (116s) 82s ago 12m 76.7M - 19.0.0-2946-g3d89accf b045ac545f82 9a0b2bf0657f 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi062.jsennv smithi062 *:80 running (2m) 82s ago 12m 76.5M - 19.0.0-2946-g3d89accf b045ac545f82 ccb5d3763116 2024-06-01T02:52:06.577 INFO:teuthology.orchestra.run.smithi062.stdout:rgw.smpl.smithi146.olokyu smithi146 *:80 running (110s) 82s ago 12m 78.4M - 19.0.0-2946-g3d89accf b045ac545f82 da2f6bd39e54 2024-06-01T02:52:07.142 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:06 smithi062 ceph-mon[64430]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:07.142 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:06 smithi062 ceph-mon[66884]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:07.165 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph versions' 2024-06-01T02:52:07.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:06 smithi146 ceph-mon[57708]: pgmap v275: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:08.013 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:07 smithi062 ceph-mon[64430]: from='client.44707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:08.014 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:07 smithi062 ceph-mon[66884]: from='client.44707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:08.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:07 smithi146 ceph-mon[57708]: from='client.44707 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:08.953 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:08 smithi062 ceph-mon[64430]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:08.953 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:08 smithi062 ceph-mon[66884]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:09.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:08 smithi146 ceph-mon[57708]: pgmap v276: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:09.285 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mon": { 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 3 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr": { 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 2 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "osd": { 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 8 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw": { 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 4 2024-06-01T02:52:09.286 INFO:teuthology.orchestra.run.smithi062.stdout: }, 2024-06-01T02:52:09.287 INFO:teuthology.orchestra.run.smithi062.stdout: "overall": { 2024-06-01T02:52:09.287 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)": 17 2024-06-01T02:52:09.287 INFO:teuthology.orchestra.run.smithi062.stdout: } 2024-06-01T02:52:09.287 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:52:09.950 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch upgrade status' 2024-06-01T02:52:10.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:09 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/3552939080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:10.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:09 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/3552939080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:10.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:09 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/3552939080' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:11.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:10 smithi146 ceph-mon[57708]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:11.214 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:10 smithi062 ceph-mon[64430]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:11.215 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:10 smithi062 ceph-mon[66884]: pgmap v277: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:11.973 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:52:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba", 2024-06-01T02:52:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: "in_progress": true, 2024-06-01T02:52:11.973 INFO:teuthology.orchestra.run.smithi062.stdout: "which": "Upgrading all daemon types on all hosts", 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "services_complete": [ 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "mon", 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "mgr", 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "osd", 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "rgw" 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: ], 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "progress": "17/23 daemons upgraded", 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "message": "Error: UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed.", 2024-06-01T02:52:11.974 INFO:teuthology.orchestra.run.smithi062.stdout: "is_paused": true 2024-06-01T02:52:11.975 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:52:12.622 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph health detail' 2024-06-01T02:52:13.184 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:13.184 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[64430]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:13.184 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[64430]: from='client.44719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:13.184 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:52:13.184 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:13.184 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[66884]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:13.185 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[66884]: from='client.44719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:13.185 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:12 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:52:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:12 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:12 smithi146 ceph-mon[57708]: pgmap v278: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:13.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:12 smithi146 ceph-mon[57708]: from='client.44719 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:13.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:12 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:HEALTH_WARN 1 failed cephadm daemon(s); Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout: daemon iscsi.foo.smithi062.ywlcmk on smithi062 is in unknown state 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:[WRN] UPGRADE_REDEPLOY_DAEMON: Upgrading daemon iscsi.foo.smithi062.ywlcmk on host smithi062 failed. 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout: Upgrade daemon: iscsi.foo.smithi062.ywlcmk: cephadm exited with an error code: 1, stderr: Redeploy daemon iscsi.foo.smithi062.ywlcmk ... 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:Creating ceph-iscsi config... 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/iscsi-gateway.cfg 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:Write file: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/iscsi.foo.smithi062.ywlcmk/tcmu-runner-entrypoint.sh 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:Failed to trim old cgroups /sys/fs/cgroup/system.slice/system-ceph\x2d176c65a6\x2d1fbf\x2d11ef\x2dbc9b\x2dc7b262605968.slice/ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service 2024-06-01T02:52:14.777 INFO:teuthology.orchestra.run.smithi062.stdout:Non-zero exit code 1 from systemctl start ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout:systemctl: stderr See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout:Traceback (most recent call last): 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: return _run_code(code, main_globals, None, 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: exec(code, run_globals) 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5618, in 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 5606, in main 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3096, in command_deploy_from 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3131, in _common_deploy 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 3151, in _deploy_daemon_container 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1087, in deploy_daemon 2024-06-01T02:52:14.778 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/__main__.py", line 1223, in deploy_daemon_units 2024-06-01T02:52:14.779 INFO:teuthology.orchestra.run.smithi062.stdout: File "/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/cephadm.8cc035ff3971b2e34a337b7a71505dc9e2ace82f3807fb6c384c059d64bde4dc/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-06-01T02:52:14.779 INFO:teuthology.orchestra.run.smithi062.stdout:RuntimeError: Failed command: systemctl start ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk: Job for ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service failed because the control process exited with error code. 2024-06-01T02:52:14.779 INFO:teuthology.orchestra.run.smithi062.stdout:See "systemctl status ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" and "journalctl -xeu ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@iscsi.foo.smithi062.ywlcmk.service" for details. 2024-06-01T02:52:14.867 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:52:14] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-06-01T02:52:15.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:14 smithi146 ceph-mon[57708]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:15.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:14 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/923186883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:52:15.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:14 smithi062 ceph-mon[64430]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:15.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:14 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/923186883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:52:15.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:14 smithi062 ceph-mon[66884]: pgmap v279: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:15.264 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:14 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/923186883' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-06-01T02:52:15.438 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-06-01T02:52:16.940 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:16 smithi062 ceph-mon[64430]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:16.940 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:16 smithi062 ceph-mon[66884]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:17.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:16 smithi146 ceph-mon[57708]: pgmap v280: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:17.453 INFO:teuthology.orchestra.run.smithi062.stdout:true 2024-06-01T02:52:18.035 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-06-01T02:52:18.059 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:17 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/2022412391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:18.060 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:17 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/2022412391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:18.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:17 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/2022412391' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:19.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:18 smithi146 ceph-mon[57708]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:19.263 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:18 smithi062 ceph-mon[64430]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:19.263 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:18 smithi062 ceph-mon[66884]: pgmap v281: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:20.224 INFO:teuthology.orchestra.run.smithi062.stdout: "ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc)" 2024-06-01T02:52:20.810 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -e sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba -- bash -c 'ceph orch ls | grep '"'"'^osd '"'"'' 2024-06-01T02:52:21.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:20 smithi062 ceph-mon[64430]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:21.166 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:20 smithi062 ceph-mon[64430]: from='client.? 172.21.15.62:0/1533993614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:21.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:20 smithi062 ceph-mon[66884]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:21.166 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:20 smithi062 ceph-mon[66884]: from='client.? 172.21.15.62:0/1533993614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:21.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:20 smithi146 ceph-mon[57708]: pgmap v282: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:21.207 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:20 smithi146 ceph-mon[57708]: from='client.? 172.21.15.62:0/1533993614' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-06-01T02:52:22.965 INFO:teuthology.orchestra.run.smithi062.stdout:osd 8 98s ago - 2024-06-01T02:52:23.250 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:23 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:23.250 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:23 smithi062 ceph-mon[64430]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:23.250 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:23 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:23.250 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:23 smithi062 ceph-mon[66884]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:23.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:23 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:23.456 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:23 smithi146 ceph-mon[57708]: pgmap v283: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:23.558 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-06-01T02:52:23.569 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi062.front.sepia.ceph.com 2024-06-01T02:52:23.570 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch upgrade ls' 2024-06-01T02:52:24.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:24 smithi062 ceph-mon[64430]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:24.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:24 smithi062 ceph-mon[66884]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:24.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:24 smithi146 ceph-mon[57708]: from='client.34812 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:24.857 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:52:24] "GET /metrics HTTP/1.1" 200 38754 "" "Prometheus/2.33.4" 2024-06-01T02:52:25.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:25 smithi062 ceph-mon[64430]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:25.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:25 smithi062 ceph-mon[66884]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:25.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:25 smithi146 ceph-mon[57708]: pgmap v284: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:26.316 INFO:teuthology.orchestra.run.smithi062.stdout:{ 2024-06-01T02:52:26.316 INFO:teuthology.orchestra.run.smithi062.stdout: "image": "quay.io/ceph/ceph", 2024-06-01T02:52:26.316 INFO:teuthology.orchestra.run.smithi062.stdout: "registry": "quay.io", 2024-06-01T02:52:26.316 INFO:teuthology.orchestra.run.smithi062.stdout: "bare_image": "ceph/ceph", 2024-06-01T02:52:26.317 INFO:teuthology.orchestra.run.smithi062.stdout: "versions": [] 2024-06-01T02:52:26.317 INFO:teuthology.orchestra.run.smithi062.stdout:} 2024-06-01T02:52:26.893 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0' 2024-06-01T02:52:27.492 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:27 smithi062 ceph-mon[64430]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:27.492 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:27 smithi062 ceph-mon[64430]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:27.493 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:27 smithi062 ceph-mon[66884]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:27.493 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:27 smithi062 ceph-mon[66884]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:27.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:27 smithi146 ceph-mon[57708]: pgmap v285: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:27.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:27 smithi146 ceph-mon[57708]: from='client.34818 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:28.450 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:28 smithi062 ceph-mon[64430]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:52:28.451 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:28 smithi062 ceph-mon[66884]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:52:28.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:28 smithi146 ceph-mon[57708]: from='mgr.34107 172.21.15.62:0/2170710738' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-01T02:52:29.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:29 smithi062 ceph-mon[64430]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:29.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:29 smithi062 ceph-mon[66884]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:29.651 INFO:teuthology.orchestra.run.smithi062.stdout: "16.2.0", 2024-06-01T02:52:29.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:29 smithi146 ceph-mon[57708]: pgmap v286: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-06-01T02:52:30.339 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2' 2024-06-01T02:52:30.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:30 smithi062 ceph-mon[64430]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:30.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:30 smithi062 ceph-mon[66884]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:30.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:30 smithi146 ceph-mon[57708]: from='client.34824 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "show_all_versions": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:31.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:31 smithi062 ceph-mon[64430]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:31.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:31 smithi062 ceph-mon[66884]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:31.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:31 smithi146 ceph-mon[57708]: pgmap v287: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:33.182 INFO:teuthology.orchestra.run.smithi062.stdout: "v16.2.2", 2024-06-01T02:52:33.182 INFO:teuthology.orchestra.run.smithi062.stdout: "v16.2.2-20210505", 2024-06-01T02:52:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:33 smithi062 ceph-mon[64430]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:33.513 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:33 smithi062 ceph-mon[64430]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:33 smithi062 ceph-mon[66884]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:33.514 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:33 smithi062 ceph-mon[66884]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:33.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:33 smithi146 ceph-mon[57708]: from='client.34704 -' entity='client.iscsi.foo.smithi062.ywlcmk' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-06-01T02:52:33.707 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:33 smithi146 ceph-mon[57708]: pgmap v288: 161 pgs: 161 active+clean; 585 KiB data, 290 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-06-01T02:52:33.807 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-01T02:52:33.817 INFO:tasks.cephadm:Teardown begin 2024-06-01T02:52:33.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:52:33.848 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:52:33.882 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-01T02:52:33.882 DEBUG:teuthology.orchestra.run.smithi062:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-01T02:52:33.906 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-01T02:52:33.940 INFO:tasks.cephadm:Stopping all daemons... 2024-06-01T02:52:33.941 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-06-01T02:52:33.941 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a 2024-06-01T02:52:34.232 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:33 smithi062 systemd[1]: Stopping Ceph mon.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:34.513 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-mon[66884]: from='client.44761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:34.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-mon[64430]: from='client.44761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:34.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a[64426]: 2024-06-01T02:52:34.257+0000 7fdd98b83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:34.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a[64426]: 2024-06-01T02:52:34.257+0000 7fdd98b83700 -1 mon.a@0(leader) e4 *** Got Signal Terminated *** 2024-06-01T02:52:34.514 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 podman[90329]: 2024-06-01 02:52:34.402040891 +0000 UTC m=+0.254807408 container died f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:52:34] ENGINE Bus STOPPING 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:52:34] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:52:34] ENGINE Bus STOPPED 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:52:34] ENGINE Bus STARTING 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: CherryPy Checker: 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: The Application mounted at '' has an empty config. 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:52:34] ENGINE Serving on http://:::9283 2024-06-01T02:52:34.515 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: [01/Jun/2024:02:52:34] ENGINE Bus STARTED 2024-06-01T02:52:34.706 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:34 smithi146 ceph-mon[57708]: from='client.44761 -' entity='client.admin' cmd=[{"prefix": "orch upgrade ls", "image": "quay.io/ceph/ceph", "tags": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-01T02:52:34.874 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: ::ffff:172.21.15.146 - - [01/Jun/2024:02:52:34] "GET /metrics HTTP/1.1" 200 38753 "" "Prometheus/2.33.4" 2024-06-01T02:52:34.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 podman[90329]: 2024-06-01 02:52:34.706162168 +0000 UTC m=+0.558928677 container cleanup f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, ceph=True, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-01T02:52:34.874 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 bash[90329]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a 2024-06-01T02:52:35.126 INFO:journalctl@ceph.mon.a.smithi062.stdout:Jun 01 02:52:34 smithi062 podman[90352]: 2024-06-01 02:52:34.873948297 +0000 UTC m=+0.470312115 container remove f8034e39fc3a9385a832e4458aaeebd0c91e322f682af37206b12b0f1cb26f00 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-a, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True) 2024-06-01T02:52:35.149 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.a.service' 2024-06-01T02:52:35.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:35.194 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-06-01T02:52:35.194 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-06-01T02:52:35.194 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.c 2024-06-01T02:52:35.510 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:35 smithi062 systemd[1]: Stopping Ceph mon.c for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:35.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c[66878]: 2024-06-01T02:52:35.508+0000 7f5a0afb7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:35.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:35 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c[66878]: 2024-06-01T02:52:35.508+0000 7f5a0afb7700 -1 mon.c@1(peon) e4 *** Got Signal Terminated *** 2024-06-01T02:52:35.764 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:35 smithi062 podman[90462]: 2024-06-01 02:52:35.531096839 +0000 UTC m=+0.145402534 container died 7ee52beadd88d9530cdc4e89d21c77d5eca51af90de3fb2fb4c4d32046189bc7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True) 2024-06-01T02:52:36.162 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:35 smithi062 podman[90462]: 2024-06-01 02:52:35.935371153 +0000 UTC m=+0.549676841 container cleanup 7ee52beadd88d9530cdc4e89d21c77d5eca51af90de3fb2fb4c4d32046189bc7 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-01T02:52:36.162 INFO:journalctl@ceph.mon.c.smithi062.stdout:Jun 01 02:52:35 smithi062 bash[90462]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-c 2024-06-01T02:52:36.297 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.c.service' 2024-06-01T02:52:36.338 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:36.338 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-06-01T02:52:36.338 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-06-01T02:52:36.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.b 2024-06-01T02:52:36.639 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:36 smithi146 systemd[1]: Stopping Ceph mon.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:36.939 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b[57704]: 2024-06-01T02:52:36.638+0000 7f1ce0be3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:36.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b[57704]: 2024-06-01T02:52:36.638+0000 7f1ce0be3700 -1 mon.b@2(peon) e4 *** Got Signal Terminated *** 2024-06-01T02:52:36.940 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:36 smithi146 podman[75617]: 2024-06-01 02:52:36.659629445 +0000 UTC m=+0.122621186 container died b8af55737ea256365dfbbb37fa45b92155a9266147813e3cf3c6f6167216baa9 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-01T02:52:37.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:37 smithi146 podman[75617]: 2024-06-01 02:52:37.122236793 +0000 UTC m=+0.585228521 container cleanup b8af55737ea256365dfbbb37fa45b92155a9266147813e3cf3c6f6167216baa9 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-01T02:52:37.206 INFO:journalctl@ceph.mon.b.smithi146.stdout:Jun 01 02:52:37 smithi146 bash[75617]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mon-b 2024-06-01T02:52:37.542 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mon.b.service' 2024-06-01T02:52:37.614 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:37.614 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-06-01T02:52:37.614 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-06-01T02:52:37.614 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y 2024-06-01T02:52:37.916 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:37 smithi062 systemd[1]: Stopping Ceph mgr.y for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:38.199 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:37 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y[60073]: 2024-06-01T02:52:37.914+0000 7f157af67700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:38.199 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:37 smithi062 podman[90585]: 2024-06-01 02:52:37.946382868 +0000 UTC m=+0.128321281 container died 65df26cb40ccddb7c473426106c2ab2412f94a4b90345324d7b0c75fac94da05 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-06-01T02:52:38.514 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:38 smithi062 podman[90585]: 2024-06-01 02:52:38.29890103 +0000 UTC m=+0.480839464 container cleanup 65df26cb40ccddb7c473426106c2ab2412f94a4b90345324d7b0c75fac94da05 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_CLEAN=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-01T02:52:38.514 INFO:journalctl@ceph.mgr.y.smithi062.stdout:Jun 01 02:52:38 smithi062 bash[90585]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-y 2024-06-01T02:52:38.795 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.y.service' 2024-06-01T02:52:38.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:38.832 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-06-01T02:52:38.832 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-06-01T02:52:38.832 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x 2024-06-01T02:52:39.184 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:52:38 smithi146 systemd[1]: Stopping Ceph mgr.x for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:39.456 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:52:39 smithi146 podman[75736]: 2024-06-01 02:52:39.183472808 +0000 UTC m=+0.154469167 container died e90d7c11bbbfb92cc9778e9b94b488efb5e1bfb02065b7da806a84e4134e34b8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528) 2024-06-01T02:52:39.820 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:52:39 smithi146 podman[75736]: 2024-06-01 02:52:39.560717394 +0000 UTC m=+0.531713751 container cleanup e90d7c11bbbfb92cc9778e9b94b488efb5e1bfb02065b7da806a84e4134e34b8 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS) 2024-06-01T02:52:39.820 INFO:journalctl@ceph.mgr.x.smithi146.stdout:Jun 01 02:52:39 smithi146 bash[75736]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-mgr-x 2024-06-01T02:52:39.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@mgr.x.service' 2024-06-01T02:52:40.027 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:40.028 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-06-01T02:52:40.028 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-01T02:52:40.028 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0 2024-06-01T02:52:40.324 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:40 smithi062 systemd[1]: Stopping Ceph osd.0 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:40.763 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:52:40.357+0000 7fc3f992d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:40.763 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:52:40.357+0000 7fc3f992d700 -1 osd.0 153 *** Got signal Terminated *** 2024-06-01T02:52:40.764 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:40 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0[70283]: 2024-06-01T02:52:40.357+0000 7fc3f992d700 -1 osd.0 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:52:46.155 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:45 smithi062 podman[90706]: 2024-06-01 02:52:45.896673561 +0000 UTC m=+5.648660304 container died fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-06-01T02:52:46.513 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:46 smithi062 podman[90706]: 2024-06-01 02:52:46.253765499 +0000 UTC m=+6.005752235 container cleanup fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-06-01T02:52:46.514 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:46 smithi062 bash[90706]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0 2024-06-01T02:52:46.893 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:46 smithi062 podman[90732]: 2024-06-01 02:52:46.54712206 +0000 UTC m=+0.644562631 container remove fa2b42ebab366b974fd71e1876d2dc219895f29f26ace3b38d0dbb9d2cbe39cb (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, ceph=True) 2024-06-01T02:52:47.201 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:46 smithi062 podman[90824]: 2024-06-01 02:52:46.795382421 +0000 UTC m=+0.017794286 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:52:47.201 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:46 smithi062 podman[90824]: 2024-06-01 02:52:46.906478305 +0000 UTC m=+0.128890219 container create b3d844831384e6434cbd2cfc4b81fbd42df75623b9ab9c19df04f242bae2e382 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True) 2024-06-01T02:52:47.201 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 podman[90824]: 2024-06-01 02:52:47.126145661 +0000 UTC m=+0.348557528 container init b3d844831384e6434cbd2cfc4b81fbd42df75623b9ab9c19df04f242bae2e382 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux ) 2024-06-01T02:52:47.201 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 podman[90824]: 2024-06-01 02:52:47.130438542 +0000 UTC m=+0.352850406 container start b3d844831384e6434cbd2cfc4b81fbd42df75623b9ab9c19df04f242bae2e382 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:52:47.201 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 podman[90824]: 2024-06-01 02:52:47.200622865 +0000 UTC m=+0.423034733 container attach b3d844831384e6434cbd2cfc4b81fbd42df75623b9ab9c19df04f242bae2e382 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-01T02:52:47.514 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 podman[90824]: 2024-06-01 02:52:47.442499473 +0000 UTC m=+0.664911334 container died b3d844831384e6434cbd2cfc4b81fbd42df75623b9ab9c19df04f242bae2e382 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:52:47.995 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-01T02:52:48.004 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 podman[90851]: 2024-06-01 02:52:47.971709738 +0000 UTC m=+0.521816716 container remove b3d844831384e6434cbd2cfc4b81fbd42df75623b9ab9c19df04f242bae2e382 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-06-01T02:52:48.004 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-01T02:52:48.004 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 systemd[1]: Stopped Ceph osd.0 for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:52:48.004 INFO:journalctl@ceph.osd.0.smithi062.stdout:Jun 01 02:52:47 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.0.service: Consumed 7.457s CPU time. 2024-06-01T02:52:48.041 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:48.041 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-01T02:52:48.041 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-01T02:52:48.042 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.1 2024-06-01T02:52:48.263 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:48 smithi062 systemd[1]: Stopping Ceph osd.1 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:48.763 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:52:48.474+0000 7f4bdc71d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:48.763 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:52:48.474+0000 7f4bdc71d700 -1 osd.1 153 *** Got signal Terminated *** 2024-06-01T02:52:48.763 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:48 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1[72546]: 2024-06-01T02:52:48.474+0000 7f4bdc71d700 -1 osd.1 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:52:54.084 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:53 smithi062 podman[90931]: 2024-06-01 02:52:53.82584151 +0000 UTC m=+5.462146997 container died 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-01T02:52:54.467 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:54 smithi062 podman[90931]: 2024-06-01 02:52:54.182559214 +0000 UTC m=+5.818864705 container cleanup 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True) 2024-06-01T02:52:54.467 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:54 smithi062 bash[90931]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1 2024-06-01T02:52:54.467 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:52:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:52:54.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:52:54.468 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:52:54 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:52:54.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:52:54.763 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:54 smithi062 podman[90949]: 2024-06-01 02:52:54.467384843 +0000 UTC m=+0.641622243 container remove 0225f93df933390032bf616a145915565527a7ef94365f6b28dd67fdd11101bc (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-01T02:52:55.088 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:54 smithi062 podman[91041]: 2024-06-01 02:52:54.818334222 +0000 UTC m=+0.109842920 container create fc1ee0725f2dee69cf4fff7e6a6a3ed529a2100da1b082344424024642ff65e3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528) 2024-06-01T02:52:55.089 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:54 smithi062 podman[91041]: 2024-06-01 02:52:54.726198739 +0000 UTC m=+0.017707441 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:52:55.395 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:55 smithi062 podman[91041]: 2024-06-01 02:52:55.088536049 +0000 UTC m=+0.380044757 container init fc1ee0725f2dee69cf4fff7e6a6a3ed529a2100da1b082344424024642ff65e3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True) 2024-06-01T02:52:55.396 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:55 smithi062 podman[91041]: 2024-06-01 02:52:55.092231503 +0000 UTC m=+0.383740198 container start fc1ee0725f2dee69cf4fff7e6a6a3ed529a2100da1b082344424024642ff65e3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-06-01T02:52:55.396 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:55 smithi062 podman[91041]: 2024-06-01 02:52:55.171339751 +0000 UTC m=+0.462848462 container attach fc1ee0725f2dee69cf4fff7e6a6a3ed529a2100da1b082344424024642ff65e3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux ) 2024-06-01T02:52:55.682 INFO:journalctl@ceph.osd.1.smithi062.stdout:Jun 01 02:52:55 smithi062 podman[91041]: 2024-06-01 02:52:55.396157684 +0000 UTC m=+0.687666380 container died fc1ee0725f2dee69cf4fff7e6a6a3ed529a2100da1b082344424024642ff65e3 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-01T02:52:55.976 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-01T02:52:56.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:52:56.017 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-01T02:52:56.017 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-01T02:52:56.017 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.2 2024-06-01T02:52:56.263 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:52:56 smithi062 systemd[1]: Stopping Ceph osd.2 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:52:56.763 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:52:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:52:56.369+0000 7f2340372700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:52:56.763 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:52:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:52:56.369+0000 7f2340372700 -1 osd.2 153 *** Got signal Terminated *** 2024-06-01T02:52:56.764 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:52:56 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2[74972]: 2024-06-01T02:52:56.369+0000 7f2340372700 -1 osd.2 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:02.014 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:01 smithi062 podman[91146]: 2024-06-01 02:53:01.728812204 +0000 UTC m=+5.485143744 container died c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-01T02:53:02.271 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91146]: 2024-06-01 02:53:02.019553602 +0000 UTC m=+5.775885122 container cleanup c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5) 2024-06-01T02:53:02.271 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 bash[91146]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2 2024-06-01T02:53:02.649 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91163]: 2024-06-01 02:53:02.271088018 +0000 UTC m=+0.536258575 container remove c4ad797c1b5a218eed5be4529e3edd3ba186a25a1792cc6a399bae37806c5f96 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:53:02.941 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91252]: 2024-06-01 02:53:02.552325335 +0000 UTC m=+0.017941180 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:53:02.941 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91252]: 2024-06-01 02:53:02.667371435 +0000 UTC m=+0.132987304 container create 99cdab3f8e66da4dbb311786c81ba491747f9672cd6838350cfd18bf7d1a6033 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-06-01T02:53:02.941 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91252]: 2024-06-01 02:53:02.857704035 +0000 UTC m=+0.323319878 container init 99cdab3f8e66da4dbb311786c81ba491747f9672cd6838350cfd18bf7d1a6033 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, ceph=True) 2024-06-01T02:53:02.941 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91252]: 2024-06-01 02:53:02.861152744 +0000 UTC m=+0.326768577 container start 99cdab3f8e66da4dbb311786c81ba491747f9672cd6838350cfd18bf7d1a6033 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, ceph=True, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:53:02.941 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:02 smithi062 podman[91252]: 2024-06-01 02:53:02.941059092 +0000 UTC m=+0.406674923 container attach 99cdab3f8e66da4dbb311786c81ba491747f9672cd6838350cfd18bf7d1a6033 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-06-01T02:53:03.263 INFO:journalctl@ceph.osd.2.smithi062.stdout:Jun 01 02:53:03 smithi062 podman[91252]: 2024-06-01 02:53:03.163354693 +0000 UTC m=+0.628970533 container died 99cdab3f8e66da4dbb311786c81ba491747f9672cd6838350cfd18bf7d1a6033 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-06-01T02:53:03.758 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-01T02:53:03.797 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:03.797 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-01T02:53:03.798 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-01T02:53:03.798 DEBUG:teuthology.orchestra.run.smithi062:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.3 2024-06-01T02:53:04.013 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:03 smithi062 systemd[1]: Stopping Ceph osd.3 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:04.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=error ts=2024-06-01T02:53:04.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:04.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:04.241Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:04.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:04.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:04.514 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:53:04.131+0000 7f5ba03be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:53:04.514 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:53:04.131+0000 7f5ba03be700 -1 osd.3 153 *** Got signal Terminated *** 2024-06-01T02:53:04.514 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:04 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:53:04.131+0000 7f5ba03be700 -1 osd.3 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:07.206 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:06 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:06.787+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:08.206 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:07 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:07.794+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:08.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:07 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:07.784+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:08.514 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:08 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3[77414]: 2024-06-01T02:53:08.157+0000 7f5b989aa700 -1 osd.3 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.625368+0000 front 2024-06-01T02:52:42.625314+0000 (oldest deadline 2024-06-01T02:53:07.925536+0000) 2024-06-01T02:53:09.206 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:08.766+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:09.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:08.853+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:09.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:08 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:08.843+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:09.915 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:09 smithi062 podman[91358]: 2024-06-01 02:53:09.621164457 +0000 UTC m=+5.590993028 container died 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528) 2024-06-01T02:53:10.199 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91358]: 2024-06-01 02:53:10.014607628 +0000 UTC m=+5.984436186 container cleanup 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-06-01T02:53:10.199 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 bash[91358]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3 2024-06-01T02:53:10.206 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:09.791+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:10.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:09.830+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:10.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:09.769+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:10.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:09 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:09.873+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:10.513 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91383]: 2024-06-01 02:53:10.199318015 +0000 UTC m=+0.577295014 container remove 2979765bc60d2edf9eeda39f55cf9448faa7260c2869f638f9a6a868f977660d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , ceph=True) 2024-06-01T02:53:10.794 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91476]: 2024-06-01 02:53:10.525409383 +0000 UTC m=+0.106506380 container create 5c31eb0da5a1f3358462f16d700b3fa9e32af9288fcfa5ed601747bd50b0f4f1 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-06-01T02:53:10.794 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91476]: 2024-06-01 02:53:10.436692968 +0000 UTC m=+0.017789969 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:53:10.795 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91476]: 2024-06-01 02:53:10.71982339 +0000 UTC m=+0.300920378 container init 5c31eb0da5a1f3358462f16d700b3fa9e32af9288fcfa5ed601747bd50b0f4f1 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-01T02:53:10.795 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91476]: 2024-06-01 02:53:10.723123139 +0000 UTC m=+0.304220118 container start 5c31eb0da5a1f3358462f16d700b3fa9e32af9288fcfa5ed601747bd50b0f4f1 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-01T02:53:10.795 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:10 smithi062 podman[91476]: 2024-06-01 02:53:10.794688865 +0000 UTC m=+0.375785843 container attach 5c31eb0da5a1f3358462f16d700b3fa9e32af9288fcfa5ed601747bd50b0f4f1 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-06-01T02:53:11.206 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:10.773+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:11.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:10.796+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:11.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:10.792+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:11.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:10 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:10.827+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:11.264 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:11 smithi062 podman[91476]: 2024-06-01 02:53:11.029792298 +0000 UTC m=+0.610889286 container died 5c31eb0da5a1f3358462f16d700b3fa9e32af9288fcfa5ed601747bd50b0f4f1 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, RELEASE=squid-3d89acc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240528) 2024-06-01T02:53:11.574 INFO:journalctl@ceph.osd.3.smithi062.stdout:Jun 01 02:53:11 smithi062 podman[91502]: 2024-06-01 02:53:11.556497604 +0000 UTC m=+0.517375827 container remove 5c31eb0da5a1f3358462f16d700b3fa9e32af9288fcfa5ed601747bd50b0f4f1 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-3-deactivate, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:53:11.579 DEBUG:teuthology.orchestra.run.smithi062:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-01T02:53:11.621 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:11.621 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-01T02:53:11.622 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-01T02:53:11.622 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.4 2024-06-01T02:53:11.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:11.747+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:11.957 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:11.787+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:11.957 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:11.869+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:11.958 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:11 smithi146 systemd[1]: Stopping Ceph osd.4 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:11.958 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:11.785+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:12.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:11.969+0000 7f86b13f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:53:12.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:11.969+0000 7f86b13f7700 -1 osd.4 153 *** Got signal Terminated *** 2024-06-01T02:53:12.456 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:11 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:11.969+0000 7f86b13f7700 -1 osd.4 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:12.956 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:12.771+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:12.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:12.719+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:12.957 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:12.820+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:12.957 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:12.905+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:12.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:12.680Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.62:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.62:8765: connect: connection refused" 2024-06-01T02:53:12.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:12.683Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.62:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.62:8765: connect: connection refused" 2024-06-01T02:53:12.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:12.684Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.62:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.62:8765: connect: connection refused" 2024-06-01T02:53:12.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:12.685Z caller=refresh.go:98 level=error component="discovery manager notify" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.62:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.62:8765: connect: connection refused" 2024-06-01T02:53:12.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:12 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:12.685Z caller=refresh.go:98 level=error component="discovery manager scrape" discovery=http msg="Unable to refresh target groups" err="Get \"http://172.21.15.62:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.62:8765: connect: connection refused" 2024-06-01T02:53:14.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:13.774+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:14.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:13.724+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:14.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:13.921+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:14.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:13 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:13.707+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:14.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=error ts=2024-06-01T02:53:14.241Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:14.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:14.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:14.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:14 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:14.242Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:15.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:14 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:14.742+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:15.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:14 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:14.757+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:15.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:14 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:14.769+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:15.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:14 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:14.940+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:15.208 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:14 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:14.940+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:16.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:15.770+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:16.207 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:15.749+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:16.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:15.986+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:16.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:15.986+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:16.208 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:15.801+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:16.208 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:15 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:15.801+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:16.956 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4[60850]: 2024-06-01T02:53:16.704+0000 7f86a99e3700 -1 osd.4 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:42.116154+0000 front 2024-06-01T02:52:42.116053+0000 (oldest deadline 2024-06-01T02:53:08.015873+0000) 2024-06-01T02:53:16.957 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:16.776+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:16.957 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:16.776+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:16.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:16.770+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:16.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:16.770+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:17.327 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:16.983+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:17.327 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:16 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:16.984+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:17.588 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:17 smithi146 podman[75860]: 2024-06-01 02:53:17.326891091 +0000 UTC m=+5.506134962 container died b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.license=GPLv2) 2024-06-01T02:53:17.957 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:17 smithi146 podman[75860]: 2024-06-01 02:53:17.753730135 +0000 UTC m=+5.932974005 container cleanup b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-01T02:53:17.957 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:17 smithi146 bash[75860]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4 2024-06-01T02:53:17.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:17.751+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:17.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:17.751+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:17.958 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:17.779+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:17.958 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:17 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:17.779+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:18.429 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:17 smithi146 podman[75874]: 2024-06-01 02:53:17.980193985 +0000 UTC m=+0.656472172 container remove b204dde364d53d49b864bb9fbca9fb356c2c57f7d0c602f38e2c21b2c85dfaf0 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-01T02:53:18.430 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:18.031+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:18.430 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:18.031+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:18.706 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:18 smithi146 podman[75959]: 2024-06-01 02:53:18.332455487 +0000 UTC m=+0.017919050 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:53:18.707 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:18 smithi146 podman[75959]: 2024-06-01 02:53:18.448034024 +0000 UTC m=+0.133497568 container create 4b3c03bff5a1fe9fde58668ac62a60568c56dada27d91e3d2f2b0eb151acbd44 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-3d89acc) 2024-06-01T02:53:18.707 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:18 smithi146 podman[75959]: 2024-06-01 02:53:18.675467138 +0000 UTC m=+0.360930695 container init 4b3c03bff5a1fe9fde58668ac62a60568c56dada27d91e3d2f2b0eb151acbd44 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528) 2024-06-01T02:53:18.707 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:18 smithi146 podman[75959]: 2024-06-01 02:53:18.679803884 +0000 UTC m=+0.365267428 container start 4b3c03bff5a1fe9fde58668ac62a60568c56dada27d91e3d2f2b0eb151acbd44 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528) 2024-06-01T02:53:18.974 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:18 smithi146 podman[75959]: 2024-06-01 02:53:18.758837571 +0000 UTC m=+0.444301117 container attach 4b3c03bff5a1fe9fde58668ac62a60568c56dada27d91e3d2f2b0eb151acbd44 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=) 2024-06-01T02:53:18.975 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:18.787+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:18.975 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:18.787+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:18.976 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:18.786+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:18.976 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:18 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:18.786+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:19.338 INFO:journalctl@ceph.osd.4.smithi146.stdout:Jun 01 02:53:18 smithi146 podman[75959]: 2024-06-01 02:53:18.974131976 +0000 UTC m=+0.659595527 container died 4b3c03bff5a1fe9fde58668ac62a60568c56dada27d91e3d2f2b0eb151acbd44 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-06-01T02:53:19.338 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:19.077+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:19.338 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:19.077+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:19.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-01T02:53:19.667 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:19.667 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-01T02:53:19.668 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-01T02:53:19.668 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.5 2024-06-01T02:53:19.878 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:19.827+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:19.878 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:19.827+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:19.878 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:19 smithi146 systemd[1]: Stopping Ceph osd.5 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:19.878 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:19.806+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:19.878 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:19 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:19.806+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:20.206 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:20.061+0000 7f2387180700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:53:20.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:20.061+0000 7f2387180700 -1 osd.5 153 *** Got signal Terminated *** 2024-06-01T02:53:20.207 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:20.061+0000 7f2387180700 -1 osd.5 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:20.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:20.038+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:20.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:20.038+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:21.063 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:20.832+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:21.063 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:20.832+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:21.063 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:20.860+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:21.063 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:20 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:20.860+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:21.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:21.062+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:21.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:21.062+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:22.109 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:21.804+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:22.109 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:21.804+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:22.109 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:21.873+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:22.109 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:21 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:21.873+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:22.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:22.108+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:22.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:22.108+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:23.151 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:22.859+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:23.151 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:22.859+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:23.151 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:22.813+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:23.152 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:22.813+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:23.152 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:22 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:22.813+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:52:59.684208+0000 front 2024-06-01T02:52:59.684190+0000 (oldest deadline 2024-06-01T02:53:21.984127+0000) 2024-06-01T02:53:23.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:23.150+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:23.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:23.150+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:24.181 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:23.772+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:24.182 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:23.772+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:24.182 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:23.772+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:52:59.684208+0000 front 2024-06-01T02:52:59.684190+0000 (oldest deadline 2024-06-01T02:53:21.984127+0000) 2024-06-01T02:53:24.182 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:23.900+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:24.182 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:23.900+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:24.182 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:23 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:23.900+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:24.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:24.180+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:24.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:24.180+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:24.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=error ts=2024-06-01T02:53:24.243Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:24.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:24.243Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:24.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:24 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:24.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:25.144 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:24.748+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:44.581617+0000 front 2024-06-01T02:52:44.581628+0000 (oldest deadline 2024-06-01T02:53:09.281582+0000) 2024-06-01T02:53:25.144 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:24.748+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:53.382749+0000 front 2024-06-01T02:52:53.382824+0000 (oldest deadline 2024-06-01T02:53:16.282662+0000) 2024-06-01T02:53:25.144 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5[63144]: 2024-06-01T02:53:24.748+0000 7f237f76c700 -1 osd.5 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:52:59.684208+0000 front 2024-06-01T02:52:59.684190+0000 (oldest deadline 2024-06-01T02:53:21.984127+0000) 2024-06-01T02:53:25.144 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:24.880+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:25.144 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:24.880+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:25.144 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:24.880+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:25.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:25 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:25.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:25.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:25 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:25.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:25.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:25 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:25.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:25.784 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:25 smithi146 podman[76064]: 2024-06-01 02:53:25.526576773 +0000 UTC m=+5.613575981 container died 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-06-01T02:53:26.112 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:25 smithi146 podman[76064]: 2024-06-01 02:53:25.966112623 +0000 UTC m=+6.053111866 container cleanup 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-3d89acc, ceph=True, GIT_BRANCH=HEAD) 2024-06-01T02:53:26.112 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:25 smithi146 bash[76064]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5 2024-06-01T02:53:26.112 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:25 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:25.885+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:26.113 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:25 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:25.885+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:26.113 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:25 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:25.885+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:26.456 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:26 smithi146 podman[76090]: 2024-06-01 02:53:26.176139099 +0000 UTC m=+0.646001468 container remove 31b032e2fbe8bc6fdd64ff210ed96e1795acca6f15f2192a242af07d3808e89d (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-06-01T02:53:26.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:26.110+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:26.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:26.111+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:26.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:26.111+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:26.863 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:26 smithi146 podman[76185]: 2024-06-01 02:53:26.489085043 +0000 UTC m=+0.017022285 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:53:26.863 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:26 smithi146 podman[76185]: 2024-06-01 02:53:26.593672798 +0000 UTC m=+0.121610021 container create 1e11a758808bf046dae1d8bffbf69b613f06eabf16badc7d96ce9e9f78e701fa (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-3d89acc, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux ) 2024-06-01T02:53:26.863 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:26 smithi146 podman[76185]: 2024-06-01 02:53:26.787742774 +0000 UTC m=+0.315680011 container init 1e11a758808bf046dae1d8bffbf69b613f06eabf16badc7d96ce9e9f78e701fa (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc) 2024-06-01T02:53:26.863 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:26 smithi146 podman[76185]: 2024-06-01 02:53:26.790912508 +0000 UTC m=+0.318849748 container start 1e11a758808bf046dae1d8bffbf69b613f06eabf16badc7d96ce9e9f78e701fa (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-01T02:53:26.863 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:26 smithi146 podman[76185]: 2024-06-01 02:53:26.862494761 +0000 UTC m=+0.390431984 container attach 1e11a758808bf046dae1d8bffbf69b613f06eabf16badc7d96ce9e9f78e701fa (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc) 2024-06-01T02:53:27.149 INFO:journalctl@ceph.osd.5.smithi146.stdout:Jun 01 02:53:27 smithi146 podman[76185]: 2024-06-01 02:53:27.092375759 +0000 UTC m=+0.620312986 container died 1e11a758808bf046dae1d8bffbf69b613f06eabf16badc7d96ce9e9f78e701fa (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8) 2024-06-01T02:53:27.149 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:26.873+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:27.149 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:26.873+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:27.150 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:26 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:26.873+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:27.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:27.148+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:27.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:27.148+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:27.586 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:27.148+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:27.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-01T02:53:27.778 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:27.778 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-01T02:53:27.778 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-06-01T02:53:27.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.6 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:27.833+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:27.833+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:27 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:27.833+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:27 smithi146 systemd[1]: Stopping Ceph osd.6 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:28.099+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:28.099+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:28.101 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:28.099+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:28.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:28.109+0000 7efc8a7db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:53:28.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:28.109+0000 7efc8a7db700 -1 osd.6 153 *** Got signal Terminated *** 2024-06-01T02:53:28.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:28.109+0000 7efc8a7db700 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:29.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:29.107+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:29.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:29.107+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:29.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:29.107+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:29.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:28.869+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:29.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:28.869+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:29.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:28 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:28.869+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:30.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:30.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:30.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:30.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:30.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:30.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:30.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:29.916+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:30.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:29.916+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:30.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:29 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:29.916+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:31.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:30.945+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:31.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:30.945+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:31.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:30 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:30.945+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:31.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:31.177+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:31.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:31.177+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:31.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:31.177+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:32.154 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:31.898+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:32.154 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:31.898+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:32.154 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:31 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:31.898+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:32.456 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:32.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:32.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:32.457 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:07.010341+0000 front 2024-06-01T02:53:07.010240+0000 (oldest deadline 2024-06-01T02:53:31.710124+0000) 2024-06-01T02:53:33.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:32.915+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:33.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:32.915+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:33.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:32 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:32.915+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:33.646 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: *** Caught signal (Segmentation fault) ** 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: in thread 7efc71da5700 thread_name:safe_timer 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc91c32d20] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x5639614bb25a] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x563961548db8] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x563961134343] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x5639613b9a61] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x5639610b0074] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x5639610b20da] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x5639610b4043] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x56396101552a] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x56396166ab15] 2024-06-01T02:53:33.647 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 11: (CommonSafeTimerThread::entry()+0x11) [0x56396166c341] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc91c281ca] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 13: clone() 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:33.306+0000 7efc71da5700 -1 *** Caught signal (Segmentation fault) ** 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: in thread 7efc71da5700 thread_name:safe_timer 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc91c32d20] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x5639614bb25a] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x563961548db8] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x563961134343] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x5639613b9a61] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x5639610b0074] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x5639610b20da] 2024-06-01T02:53:33.648 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x5639610b4043] 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x56396101552a] 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x56396166ab15] 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 11: (CommonSafeTimerThread::entry()+0x11) [0x56396166c341] 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc91c281ca] 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 13: clone() 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -5885> 2024-06-01T02:53:06.787+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -5769> 2024-06-01T02:53:07.794+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -5153> 2024-06-01T02:53:08.843+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -4940> 2024-06-01T02:53:09.873+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -4827> 2024-06-01T02:53:10.827+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.649 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -4670> 2024-06-01T02:53:11.869+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -4574> 2024-06-01T02:53:12.905+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3967> 2024-06-01T02:53:13.921+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3808> 2024-06-01T02:53:14.940+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3807> 2024-06-01T02:53:14.940+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3639> 2024-06-01T02:53:15.986+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3638> 2024-06-01T02:53:15.986+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3078> 2024-06-01T02:53:16.983+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -3077> 2024-06-01T02:53:16.984+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2943> 2024-06-01T02:53:18.031+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.650 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2942> 2024-06-01T02:53:18.031+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.651 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2769> 2024-06-01T02:53:19.077+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.651 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2768> 2024-06-01T02:53:19.077+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.651 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2622> 2024-06-01T02:53:20.038+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.651 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2621> 2024-06-01T02:53:20.038+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.651 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2517> 2024-06-01T02:53:21.062+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2516> 2024-06-01T02:53:21.062+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2090> 2024-06-01T02:53:22.108+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2089> 2024-06-01T02:53:22.108+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2005> 2024-06-01T02:53:23.150+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -2004> 2024-06-01T02:53:23.150+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1493> 2024-06-01T02:53:24.180+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1492> 2024-06-01T02:53:24.180+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1397> 2024-06-01T02:53:25.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1396> 2024-06-01T02:53:25.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1395> 2024-06-01T02:53:25.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1257> 2024-06-01T02:53:26.110+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1256> 2024-06-01T02:53:26.111+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1255> 2024-06-01T02:53:26.111+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.652 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1185> 2024-06-01T02:53:27.148+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1184> 2024-06-01T02:53:27.148+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1183> 2024-06-01T02:53:27.148+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1154> 2024-06-01T02:53:28.099+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1153> 2024-06-01T02:53:28.099+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1152> 2024-06-01T02:53:28.099+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1115> 2024-06-01T02:53:28.109+0000 7efc8a7db700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1114> 2024-06-01T02:53:28.109+0000 7efc8a7db700 -1 osd.6 153 *** Got signal Terminated *** 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1112> 2024-06-01T02:53:28.109+0000 7efc8a7db700 -1 osd.6 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1038> 2024-06-01T02:53:29.107+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1037> 2024-06-01T02:53:29.107+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -1036> 2024-06-01T02:53:29.107+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -771> 2024-06-01T02:53:30.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.653 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -770> 2024-06-01T02:53:30.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -769> 2024-06-01T02:53:30.143+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -642> 2024-06-01T02:53:31.177+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -641> 2024-06-01T02:53:31.177+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -640> 2024-06-01T02:53:31.177+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -207> 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:40.806616+0000 front 2024-06-01T02:52:40.806488+0000 (oldest deadline 2024-06-01T02:53:06.706430+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -206> 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:48.908004+0000 front 2024-06-01T02:52:48.907934+0000 (oldest deadline 2024-06-01T02:53:14.807873+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -205> 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.708864+0000 front 2024-06-01T02:53:00.708935+0000 (oldest deadline 2024-06-01T02:53:24.208874+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: -204> 2024-06-01T02:53:32.153+0000 7efc82dc7700 -1 osd.6 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:07.010341+0000 front 2024-06-01T02:53:07.010240+0000 (oldest deadline 2024-06-01T02:53:31.710124+0000) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 0> 2024-06-01T02:53:33.306+0000 7efc71da5700 -1 *** Caught signal (Segmentation fault) ** 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: in thread 7efc71da5700 thread_name:safe_timer 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: ceph version 19.0.0-2946-g3d89accf (3d89accfe3201286f000e065a1faea2a88c7a6ba) squid (rc) 2024-06-01T02:53:33.654 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7efc91c32d20] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2: (BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, std::__cxx11::list >*, boost::intrusive_ptr)+0x40a) [0x5639614bb25a] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 3: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0x1f8) [0x563961548db8] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 4: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x53) [0x563961134343] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 5: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr >&&, eversion_t const&, eversion_t const&, std::vector >&&, std::optional&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr)+0x7b1) [0x5639613b9a61] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 6: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x424) [0x5639610b0074] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 7: (PrimaryLogPG::simple_opc_submit(std::unique_ptr >)+0x5a) [0x5639610b20da] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 8: (PrimaryLogPG::handle_watch_timeout(std::shared_ptr)+0x873) [0x5639610b4043] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 9: (HandleWatchTimeout::complete(int)+0x11a) [0x56396101552a] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 10: (CommonSafeTimer::timer_thread()+0x155) [0x56396166ab15] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 11: (CommonSafeTimerThread::entry()+0x11) [0x56396166c341] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7efc91c281ca] 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 13: clone() 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-06-01T02:53:33.655 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6[65448]: 2024-06-01T02:53:33.656 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 podman[76291]: 2024-06-01 02:53:33.386692937 +0000 UTC m=+5.380546922 container died 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-01T02:53:33.903 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:33.880+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:33.903 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:33.880+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:33.903 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:33 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:33.880+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:34.206 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 podman[76291]: 2024-06-01 02:53:33.903104503 +0000 UTC m=+5.896958530 container cleanup 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc) 2024-06-01T02:53:34.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:33 smithi146 bash[76291]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6 2024-06-01T02:53:34.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76306]: 2024-06-01 02:53:34.121153596 +0000 UTC m=+0.730306418 container remove 5a8179547bdeb68a561652048a868e5bfd76048f1efd998f2abdb2059bd82960 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-01T02:53:34.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.6.service: Main process exited, code=exited, status=139/n/a 2024-06-01T02:53:34.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=error ts=2024-06-01T02:53:34.244Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:34.513 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:34.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:34.514 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:34 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:34.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:34.706 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76398]: 2024-06-01 02:53:34.351308184 +0000 UTC m=+0.017525958 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:53:34.707 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76398]: 2024-06-01 02:53:34.455154388 +0000 UTC m=+0.121372147 container create e213e1301bd0ffc39524f15c82ba79188f365bd8e0be3ab554d27e632235f0bf (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_CLEAN=True, org.label-schema.build-date=20240528) 2024-06-01T02:53:34.707 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76398]: 2024-06-01 02:53:34.649191042 +0000 UTC m=+0.315408795 container init e213e1301bd0ffc39524f15c82ba79188f365bd8e0be3ab554d27e632235f0bf (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, RELEASE=squid-3d89acc, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-06-01T02:53:34.707 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76398]: 2024-06-01 02:53:34.65260079 +0000 UTC m=+0.318818553 container start e213e1301bd0ffc39524f15c82ba79188f365bd8e0be3ab554d27e632235f0bf (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.schema-version=1.0, RELEASE=squid-3d89acc) 2024-06-01T02:53:35.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:34 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:34.855+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:35.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:34 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:34.855+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:35.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:34 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:34.855+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:35.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76398]: 2024-06-01 02:53:34.724085583 +0000 UTC m=+0.390303358 container attach e213e1301bd0ffc39524f15c82ba79188f365bd8e0be3ab554d27e632235f0bf (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc) 2024-06-01T02:53:35.207 INFO:journalctl@ceph.osd.6.smithi146.stdout:Jun 01 02:53:34 smithi146 podman[76398]: 2024-06-01 02:53:34.952903334 +0000 UTC m=+0.619121095 container died e213e1301bd0ffc39524f15c82ba79188f365bd8e0be3ab554d27e632235f0bf (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-3d89acc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, ceph=True) 2024-06-01T02:53:35.535 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.6.service' 2024-06-01T02:53:35.575 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:35.575 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-06-01T02:53:35.576 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-06-01T02:53:35.576 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.7 2024-06-01T02:53:35.838 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 systemd[1]: Stopping Ceph osd.7 for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.837+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.837+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.837+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.837+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:08.969082+0000 front 2024-06-01T02:53:08.969091+0000 (oldest deadline 2024-06-01T02:53:34.869100+0000) 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.947+0000 7f2c997be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.947+0000 7f2c997be700 -1 osd.7 153 *** Got signal Terminated *** 2024-06-01T02:53:36.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:35 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:35.947+0000 7f2c997be700 -1 osd.7 153 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-01T02:53:37.108 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:36.854+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:37.108 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:36.854+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:37.108 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:36.854+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:37.108 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:36 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:36.854+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:08.969082+0000 front 2024-06-01T02:53:08.969091+0000 (oldest deadline 2024-06-01T02:53:34.869100+0000) 2024-06-01T02:53:38.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:37.866+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:38.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:37.866+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:38.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:37.866+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:38.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:37 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:37.866+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:08.969082+0000 front 2024-06-01T02:53:08.969091+0000 (oldest deadline 2024-06-01T02:53:34.869100+0000) 2024-06-01T02:53:39.206 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:38.852+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:39.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:38.852+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:39.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:38.852+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:39.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:38 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:38.852+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:08.969082+0000 front 2024-06-01T02:53:08.969091+0000 (oldest deadline 2024-06-01T02:53:34.869100+0000) 2024-06-01T02:53:40.206 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:39.872+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:40.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:39.872+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:40.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:39.872+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:40.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:39 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:39.872+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:08.969082+0000 front 2024-06-01T02:53:08.969091+0000 (oldest deadline 2024-06-01T02:53:34.869100+0000) 2024-06-01T02:53:41.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:40.863+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6806 osd.0 since back 2024-06-01T02:52:43.365342+0000 front 2024-06-01T02:52:43.365217+0000 (oldest deadline 2024-06-01T02:53:07.465177+0000) 2024-06-01T02:53:41.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:40.863+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6814 osd.1 since back 2024-06-01T02:52:52.666711+0000 front 2024-06-01T02:52:52.666636+0000 (oldest deadline 2024-06-01T02:53:15.566675+0000) 2024-06-01T02:53:41.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:40.863+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6822 osd.2 since back 2024-06-01T02:53:00.767982+0000 front 2024-06-01T02:53:00.768007+0000 (oldest deadline 2024-06-01T02:53:23.067855+0000) 2024-06-01T02:53:41.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:40.863+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.62:6830 osd.3 since back 2024-06-01T02:53:08.969082+0000 front 2024-06-01T02:53:08.969091+0000 (oldest deadline 2024-06-01T02:53:34.869100+0000) 2024-06-01T02:53:41.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:40 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7[67766]: 2024-06-01T02:53:40.863+0000 7f2c95db2700 -1 osd.7 153 heartbeat_check: no reply from 172.21.15.146:6804 osd.4 since back 2024-06-01T02:53:14.869618+0000 front 2024-06-01T02:53:14.869493+0000 (oldest deadline 2024-06-01T02:53:40.169495+0000) 2024-06-01T02:53:41.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:41 smithi146 podman[76503]: 2024-06-01 02:53:41.464192916 +0000 UTC m=+5.657795830 container died 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528) 2024-06-01T02:53:41.797 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:41 smithi146 podman[76503]: 2024-06-01 02:53:41.797137452 +0000 UTC m=+5.990740378 container cleanup 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , RELEASE=squid-3d89acc, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-01T02:53:42.206 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:41 smithi146 bash[76503]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7 2024-06-01T02:53:42.207 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:41 smithi146 podman[76533]: 2024-06-01 02:53:41.973414554 +0000 UTC m=+0.506342764 container remove 18402d72526215ee4a499833c3ef88a66721e68344b5500daed6f489329bc27f (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, RELEASE=squid-3d89acc, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-06-01T02:53:42.669 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:42 smithi146 podman[76622]: 2024-06-01 02:53:42.407878993 +0000 UTC m=+0.107621586 container create ffe35fc2df0c0d3aee8780e95224152956f8656b23178364b11a9bd221160888 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-3d89acc, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, org.label-schema.build-date=20240528) 2024-06-01T02:53:42.669 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:42 smithi146 podman[76622]: 2024-06-01 02:53:42.317937209 +0000 UTC m=+0.017679808 image pull b045ac545f826a15724cc02d20333d2cc28f7e0eeb5a24d748d09a02280116e1 quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-01T02:53:42.669 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:42 smithi146 podman[76622]: 2024-06-01 02:53:42.669154913 +0000 UTC m=+0.368897508 container init ffe35fc2df0c0d3aee8780e95224152956f8656b23178364b11a9bd221160888 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, CEPH_POINT_RELEASE=, ceph=True) 2024-06-01T02:53:42.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:42 smithi146 podman[76622]: 2024-06-01 02:53:42.673338102 +0000 UTC m=+0.373080696 container start ffe35fc2df0c0d3aee8780e95224152956f8656b23178364b11a9bd221160888 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-3d89acc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-06-01T02:53:42.957 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:42 smithi146 podman[76622]: 2024-06-01 02:53:42.760076333 +0000 UTC m=+0.459818920 container attach ffe35fc2df0c0d3aee8780e95224152956f8656b23178364b11a9bd221160888 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, RELEASE=squid-3d89acc, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240528, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-06-01T02:53:43.296 INFO:journalctl@ceph.osd.7.smithi146.stdout:Jun 01 02:53:42 smithi146 podman[76622]: 2024-06-01 02:53:42.967792518 +0000 UTC m=+0.667535104 container died ffe35fc2df0c0d3aee8780e95224152956f8656b23178364b11a9bd221160888 (image=quay.ceph.io/ceph-ci/ceph:3d89accfe3201286f000e065a1faea2a88c7a6ba, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-osd-7-deactivate, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-3d89acc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=fe3baff630f29ddc3eb9757f6b798ccb55ab7bd8, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-01T02:53:43.586 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@osd.7.service' 2024-06-01T02:53:43.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:43.626 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-06-01T02:53:43.626 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-06-01T02:53:43.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a 2024-06-01T02:53:43.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 systemd[1]: Stopping Ceph prometheus.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:43.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.799Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:53:43.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-06-01T02:53:43.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-06-01T02:53:43.957 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a[53656]: ts=2024-06-01T02:53:43.800Z caller=main.go:1066 level=info msg="See you next time!" 2024-06-01T02:53:43.958 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:43 smithi146 podman[76701]: 2024-06-01 02:53:43.810400374 +0000 UTC m=+0.097307606 container died 2565089ff6fe0382e77cea2a25581ebb3118bf6e22042c34f767bb2c219c7133 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:44.426 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:44 smithi146 podman[76701]: 2024-06-01 02:53:44.166150172 +0000 UTC m=+0.453057403 container cleanup 2565089ff6fe0382e77cea2a25581ebb3118bf6e22042c34f767bb2c219c7133 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:44.426 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:44 smithi146 bash[76701]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus-a 2024-06-01T02:53:44.426 INFO:journalctl@ceph.prometheus.a.smithi146.stdout:Jun 01 02:53:44 smithi146 bash[76727]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-prometheus.a" found: no such container 2024-06-01T02:53:44.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo pkill -f 'journalctl -f -n 0 -u ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@prometheus.a.service' 2024-06-01T02:53:44.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-01T02:53:44.467 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-06-01T02:53:44.467 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-01T02:53:44.495 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=error ts=2024-06-01T02:53:44.244Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:44.495 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:44.244Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi062.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.62:8443: connect: connection refused" 2024-06-01T02:53:44.495 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:44 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=warn ts=2024-06-01T02:53:44.245Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi146.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.146:8443: connect: connection refused" 2024-06-01T02:53:46.434 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 systemd[1]: Stopping Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:46.435 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 podman[91840]: 2024-06-01 02:53:46.193141149 +0000 UTC m=+0.110496325 container died eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:46.710 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 podman[91840]: 2024-06-01 02:53:46.507618586 +0000 UTC m=+0.424973768 container cleanup eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:46.710 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 bash[91840]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a 2024-06-01T02:53:46.710 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 podman[91854]: 2024-06-01 02:53:46.667130218 +0000 UTC m=+0.477349203 container remove eb8c079db15008a9128b0fd9adaad25999a8ff4bd34cbe8a449f687bc668ce45 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:46.710 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:53:47.014 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:46 smithi062 systemd[1]: Stopping Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:53:47.014 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 bash[91868]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter.a" found: no such container 2024-06-01T02:53:47.014 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-06-01T02:53:47.014 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 systemd[1]: Stopped Ceph node-exporter.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:53:47.015 INFO:journalctl@ceph.node-exporter.a.smithi062.stdout:Jun 01 02:53:46 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.a.service: Consumed 3.896s CPU time. 2024-06-01T02:53:47.330 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a[59029]: level=info ts=2024-06-01T02:53:47.041Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-06-01T02:53:47.330 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 podman[91919]: 2024-06-01 02:53:47.092636462 +0000 UTC m=+0.155367749 container died f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:47.630 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 podman[91919]: 2024-06-01 02:53:47.420179514 +0000 UTC m=+0.482910791 container cleanup f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:47.630 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 bash[91919]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a 2024-06-01T02:53:47.990 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 podman[91931]: 2024-06-01 02:53:47.629893308 +0000 UTC m=+0.538052726 container remove f2f8ba65c5233ceb9b285284295905f644e154f123728bd97da1f777d72a51e4 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-06-01T02:53:47.990 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 podman[91931]: 2024-06-01 02:53:47.705486926 +0000 UTC m=+0.613646352 volume remove d0383d4fe8c5b86e86112a7b864a656e5d8f84f54891383170f084746f457c17 2024-06-01T02:53:47.990 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 bash[91945]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-alertmanager.a" found: no such container 2024-06-01T02:53:47.990 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-06-01T02:53:47.990 INFO:journalctl@ceph.alertmanager.a.smithi062.stdout:Jun 01 02:53:47 smithi062 systemd[1]: Stopped Ceph alertmanager.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:54:21.751 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-01T02:54:23.452 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:23 smithi146 systemd[1]: Stopping Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:54:23.706 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:23 smithi146 podman[77064]: 2024-06-01 02:54:23.451739803 +0000 UTC m=+0.120106564 container died c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:54:24.084 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:23 smithi146 podman[77064]: 2024-06-01 02:54:23.799524007 +0000 UTC m=+0.467890765 container cleanup c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:54:24.084 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:23 smithi146 bash[77064]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b 2024-06-01T02:54:24.084 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:23 smithi146 bash[77092]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter.b" found: no such container 2024-06-01T02:54:24.429 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:24 smithi146 podman[77076]: 2024-06-01 02:54:24.083902624 +0000 UTC m=+0.635682649 container remove c725f8ac02c4315c1eb11a4ca215bc1db87f9aeca55d54b12cb189465d26d52e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-06-01T02:54:24.430 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-06-01T02:54:24.430 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-06-01T02:54:24.430 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: Stopped Ceph node-exporter.b for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:54:24.430 INFO:journalctl@ceph.node-exporter.b.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@node-exporter.b.service: Consumed 3.753s CPU time. 2024-06-01T02:54:24.430 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: Stopping Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968... 2024-06-01T02:54:24.696 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a[50924]: t=2024-06-01T02:54:24+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-06-01T02:54:24.696 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 podman[77147]: 2024-06-01 02:54:24.439595766 +0000 UTC m=+0.153652746 container died 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, vendor=Red Hat, Inc., version=8.5, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, architecture=x86_64) 2024-06-01T02:54:24.956 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 podman[77147]: 2024-06-01 02:54:24.778678794 +0000 UTC m=+0.492735765 container cleanup 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64, description=Ceph Grafana Container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, version=8.5, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc.) 2024-06-01T02:54:24.957 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 bash[77147]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a 2024-06-01T02:54:25.433 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 podman[77160]: 2024-06-01 02:54:24.963047313 +0000 UTC m=+0.522331807 container remove 138c911bd936c3f927e57adc62152cb85582127dfe1825a012fb8e66c880d6d9 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, architecture=x86_64, vcs-type=git, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.24.2, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-03-28T10:36:18.413762, name=ubi8) 2024-06-01T02:54:25.433 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 bash[77174]: Error: no container with name or ID "ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968-grafana.a" found: no such container 2024-06-01T02:54:25.434 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@grafana.a.service: Deactivated successfully. 2024-06-01T02:54:25.434 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:24 smithi146 systemd[1]: Stopped Ceph grafana.a for 176c65a6-1fbf-11ef-bc9b-c7b262605968. 2024-06-01T02:54:25.434 INFO:journalctl@ceph.grafana.a.smithi146.stdout:Jun 01 02:54:25 smithi146 systemd[1]: ceph-176c65a6-1fbf-11ef-bc9b-c7b262605968@grafana.a.service: Consumed 4.000s CPU time. 2024-06-01T02:54:47.963 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:54:47.999 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-01T02:54:48.026 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-01T02:54:48.028 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/remote/smithi062/crash 2024-06-01T02:54:48.029 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/crash -- . 2024-06-01T02:54:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:tar: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-01T02:54:48.082 INFO:teuthology.orchestra.run.smithi062.stderr:tar: Error is not recoverable: exiting now 2024-06-01T02:54:48.084 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/remote/smithi146/crash 2024-06-01T02:54:48.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/crash -- . 2024-06-01T02:54:48.121 INFO:teuthology.orchestra.run.smithi146.stderr:tar: /var/lib/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-01T02:54:48.121 INFO:teuthology.orchestra.run.smithi146.stderr:tar: Error is not recoverable: exiting now 2024-06-01T02:54:48.122 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-01T02:54:48.122 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-01T02:54:48.154 INFO:teuthology.orchestra.run.smithi062.stdout:2024-06-01T02:44:06.972713+0000 mon.a (mon.0) 977 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:54:48.155 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-01T02:54:48.155 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[SEC\]' /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-01T02:54:48.224 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[ERR\]' /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-01T02:54:48.295 DEBUG:teuthology.orchestra.run.smithi062:> sudo egrep '\[WRN\]' /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-06-01T02:54:48.363 INFO:teuthology.orchestra.run.smithi062.stdout:2024-06-01T02:44:06.972713+0000 mon.a (mon.0) 977 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-06-01T02:54:48.363 INFO:tasks.cephadm:Compressing logs... 2024-06-01T02:54:48.363 DEBUG:teuthology.orchestra.run.smithi062:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-01T02:54:48.406 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-01T02:54:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-01T02:54:48.429 INFO:teuthology.orchestra.run.smithi062.stderr:: No such file or directory 2024-06-01T02:54:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.smpl.smithi062.jsennv.log 2024-06-01T02:54:48.430 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log 2024-06-01T02:54:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.smpl.smithi062.jsennv.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-01T02:54:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log: 76.0% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.smpl.smithi062.jsennv.log.gz 2024-06-01T02:54:48.431 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-01T02:54:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.foo.smithi062.lqlihd.log 2024-06-01T02:54:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-01T02:54:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.foo.smithi062.lqlihd.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-01T02:54:48.435 INFO:teuthology.orchestra.run.smithi062.stderr: 76.5% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.foo.smithi062.lqlihd.log.gz 2024-06-01T02:54:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-06-01T02:54:48.435 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-01T02:54:48.436 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-06-01T02:54:48.436 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.0.log: 93.0% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-01T02:54:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-01T02:54:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:: No such file or directory 2024-06-01T02:54:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.foo.smithi146.btivlw.log 2024-06-01T02:54:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log 2024-06-01T02:54:48.438 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.foo.smithi146.btivlw.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-01T02:54:48.438 INFO:teuthology.orchestra.run.smithi146.stderr: 76.4% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.foo.smithi146.btivlw.log.gz 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.smpl.smithi146.olokyu.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr: 87.5% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.log.gz/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.6.log: 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.smpl.smithi146.olokyu.log: 76.3% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-client.rgw.smpl.smithi146.olokyu.log.gz 2024-06-01T02:54:48.439 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-01T02:54:48.440 INFO:teuthology.orchestra.run.smithi062.stderr: 94.2% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-01T02:54:48.440 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-06-01T02:54:48.440 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-01T02:54:48.442 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.a.log: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-01T02:54:48.443 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-01T02:54:48.444 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-06-01T02:54:48.445 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-volume.log: 83.7% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-01T02:54:48.451 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.b.log: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-01T02:54:48.452 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/tcmu-runner.log 2024-06-01T02:54:48.454 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.cephadm.log: 90.8% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-01T02:54:48.454 INFO:teuthology.orchestra.run.smithi062.stderr:gzip -5 --verbose -- /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-01T02:54:48.455 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/tcmu-runner.log: 84.6% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/tcmu-runner.log.gz 2024-06-01T02:54:48.457 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-01T02:54:48.469 INFO:teuthology.orchestra.run.smithi062.stderr:/var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-01T02:54:48.483 INFO:teuthology.orchestra.run.smithi146.stderr: 89.6% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-06-01T02:54:48.774 INFO:teuthology.orchestra.run.smithi146.stderr: 92.4% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-06-01T02:54:48.790 INFO:teuthology.orchestra.run.smithi062.stderr: 88.6% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-06-01T02:54:48.931 INFO:teuthology.orchestra.run.smithi062.stderr: 92.2% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-06-01T02:54:49.331 INFO:teuthology.orchestra.run.smithi062.stderr: 93.6% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-01T02:54:49.532 INFO:teuthology.orchestra.run.smithi062.stderr: 91.0% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-06-01T02:54:49.597 INFO:teuthology.orchestra.run.smithi146.stderr: 93.5% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-06-01T02:54:49.721 INFO:teuthology.orchestra.run.smithi062.stderr: 93.7% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-01T02:54:49.754 INFO:teuthology.orchestra.run.smithi146.stderr: 93.7% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-01T02:54:49.866 INFO:teuthology.orchestra.run.smithi146.stderr: 94.0% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-06-01T02:54:50.106 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-01T02:54:50.108 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-06-01T02:54:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m1.687s 2024-06-01T02:54:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m5.798s 2024-06-01T02:54:50.108 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.229s 2024-06-01T02:54:50.189 INFO:teuthology.orchestra.run.smithi062.stderr: 93.7% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-01T02:54:50.252 INFO:teuthology.orchestra.run.smithi062.stderr: 93.7% -- replaced with /var/log/ceph/176c65a6-1fbf-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-01T02:54:50.253 INFO:teuthology.orchestra.run.smithi062.stderr: 2024-06-01T02:54:50.254 INFO:teuthology.orchestra.run.smithi062.stderr:real 0m1.834s 2024-06-01T02:54:50.254 INFO:teuthology.orchestra.run.smithi062.stderr:user 0m7.496s 2024-06-01T02:54:50.254 INFO:teuthology.orchestra.run.smithi062.stderr:sys 0m0.265s 2024-06-01T02:54:50.254 INFO:tasks.cephadm:Archiving logs... 2024-06-01T02:54:50.254 DEBUG:teuthology.misc:Transferring archived files from smithi062:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/remote/smithi062/log 2024-06-01T02:54:50.256 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-01T02:54:51.245 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/remote/smithi146/log 2024-06-01T02:54:51.246 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-01T02:54:52.034 INFO:tasks.cephadm:Removing cluster... 2024-06-01T02:54:52.034 DEBUG:teuthology.orchestra.run.smithi062:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 --force 2024-06-01T02:54:52.327 DEBUG:teuthology.orchestra.run.smithi146:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 176c65a6-1fbf-11ef-bc9b-c7b262605968 --force 2024-06-01T02:54:52.633 INFO:tasks.cephadm:Removing cephadm ... 2024-06-01T02:54:52.634 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-01T02:54:52.650 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-01T02:54:52.666 INFO:tasks.cephadm:Teardown complete 2024-06-01T02:54:52.666 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-01T02:54:52.679 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-01T02:54:52.679 DEBUG:teuthology.orchestra.run.smithi062:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T02:54:52.693 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-01T02:54:52.705 INFO:teuthology.orchestra.run.smithi062.stderr:bash: line 1: ntpq: command not found 2024-06-01T02:54:52.709 INFO:teuthology.orchestra.run.smithi062.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T02:54:52.709 INFO:teuthology.orchestra.run.smithi062.stdout:=============================================================================== 2024-06-01T02:54:52.710 INFO:teuthology.orchestra.run.smithi062.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 +853us[ +864us] +/- 45ms 2024-06-01T02:54:52.710 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 19 -1329us[-1318us] +/- 60ms 2024-06-01T02:54:52.710 INFO:teuthology.orchestra.run.smithi062.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +195us[ +206us] +/- 67ms 2024-06-01T02:54:52.710 INFO:teuthology.orchestra.run.smithi062.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:54:52.720 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-06-01T02:54:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-01T02:54:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-06-01T02:54:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv01.front.sepia.ceph.com 3 6 377 21 +858us[ +851us] +/- 45ms 2024-06-01T02:54:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 87 -1329us[-1347us] +/- 59ms 2024-06-01T02:54:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +221us[ +215us] +/- 67ms 2024-06-01T02:54:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-01T02:54:52.725 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-01T02:54:52.735 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-01T02:54:52.735 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-01T02:54:52.744 DEBUG:teuthology.orchestra.run.smithi062:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-01T02:54:52.830 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-01T02:54:52.904 DEBUG:teuthology.orchestra.run.smithi062:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-01T02:54:52.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:54:52.933 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-01T02:54:52.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:54:52.961 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-01T02:54:52.971 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-01T02:54:52.980 INFO:teuthology.task.internal:Duration was 1653.987663 seconds 2024-06-01T02:54:52.981 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-01T02:54:52.989 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-01T02:54:52.989 DEBUG:teuthology.orchestra.run.smithi062:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-01T02:54:52.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-01T02:54:53.029 INFO:teuthology.orchestra.run.smithi062.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T02:54:53.041 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-01T02:54:53.243 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-01T02:54:53.243 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi062.front.sepia.ceph.com 2024-06-01T02:54:53.243 DEBUG:teuthology.orchestra.run.smithi062:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-01T02:54:53.265 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-06-01T02:54:53.265 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-01T02:54:53.312 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-01T02:54:53.312 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-01T02:54:53.315 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-01T02:54:53.379 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-01T02:54:53.380 DEBUG:teuthology.orchestra.run.smithi062:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-01T02:54:53.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-01T02:54:53.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-01T02:54:53.817 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-01T02:54:53.818 DEBUG:teuthology.orchestra.run.smithi062:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-01T02:54:53.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-01T02:54:53.868 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-01T02:54:53.878 DEBUG:teuthology.orchestra.run.smithi062:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-01T02:54:53.886 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-01T02:54:53.907 INFO:teuthology.orchestra.run.smithi062.stdout:kernel.core_pattern = core 2024-06-01T02:54:53.931 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-06-01T02:54:53.959 DEBUG:teuthology.orchestra.run.smithi062:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-01T02:54:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:54:53.989 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-01T02:54:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-01T02:54:54.014 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-01T02:54:54.025 INFO:teuthology.task.internal:Transferring archived files... 2024-06-01T02:54:54.026 DEBUG:teuthology.misc:Transferring archived files from smithi062:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/remote/smithi062 2024-06-01T02:54:54.026 DEBUG:teuthology.orchestra.run.smithi062:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-01T02:54:54.075 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-31_21:08:02-orch-squid-distro-default-smithi/7736417/remote/smithi146 2024-06-01T02:54:54.075 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-01T02:54:54.113 INFO:teuthology.task.internal:Removing archive directory... 2024-06-01T02:54:54.113 DEBUG:teuthology.orchestra.run.smithi062:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-01T02:54:54.116 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-01T02:54:54.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-01T02:54:54.173 INFO:teuthology.task.internal:Not uploading archives. 2024-06-01T02:54:54.173 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-01T02:54:54.182 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-01T02:54:54.183 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-01T02:54:54.185 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-01T02:54:54.198 INFO:teuthology.orchestra.run.smithi062.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 1 02:54 /home/ubuntu/cephtest 2024-06-01T02:54:54.216 INFO:teuthology.orchestra.run.smithi146.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 1 02:54 /home/ubuntu/cephtest 2024-06-01T02:54:54.217 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-01T02:54:54.227 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-01T02:54:54.249 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 1653.9876627922058 failure_reason: '"2024-06-01T02:44:06.972713+0000 mon.a (mon.0) 977 : cluster [WRN] Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON)" in cluster log' owner: scheduled_teuthology@teuthology success: false 2024-06-01T02:54:54.250 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-01T02:54:54.333 INFO:teuthology.run:FAIL